{"full_name":"pathwaycom/pathway","owner":"pathwaycom","name":"pathway","shard":"1","url":"https://github.com/pathwaycom/pathway","description":"Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.","homepage":"https://pathway.com","topics":["batch-processing","kafka","pathway","python","streaming","machine-learning-algorithms","real-time","data-analytics","data-pipelines","data-processing","dataflow","etl","etl-framework","iot-analytics","rust"],"tag_pages":["data-analytics","data-pipelines","etl","kafka","machine-learning-algorithms","python","real-time","rust","streaming"],"language":"Python","license":"NOASSERTION","avatar":"https://avatars.githubusercontent.com/u/25750857?v=4","created_at":"2022-11-27T13:01:14Z","pushed_at":"2026-11-15T06:51:06Z","metrics":{"stars":61994,"forks":1706,"watchers":115,"issues":175,"prs":77,"releases":89,"issues_open":30,"issues_closed":145,"prs_open":7,"prs_closed":37,"prs_merged":33,"discussions":0},"growth":{"day":{"stars":0,"forks":-1,"base_day":"2026-11-15"},"week":{"stars":-18,"forks":-2,"base_day":"2026-11-09"},"month":{"stars":-116,"forks":23,"base_day":"2026-10-17"}},"history":{"dates":["2026-09-06","2026-09-07","2026-09-08","2026-09-09","2026-09-10","2026-09-11","2026-09-12","2026-09-13","2026-09-14","2026-09-15","2026-09-16","2026-09-17","2026-09-18","2026-09-19","2026-09-20","2026-09-21","2026-09-22","2026-09-23","2026-09-24","2026-09-25","2026-09-26","2026-09-27","2026-09-28","2026-09-29","2026-09-30","2026-10-01","2026-10-02","2026-10-03","2026-10-04","2026-10-05","2026-10-06","2026-10-07","2026-10-08","2026-10-09","2026-10-10","2026-10-11","2026-10-12","2026-10-13","2026-10-14","2026-10-15","2026-10-16","2026-10-17","2026-10-18","2026-10-19","2026-10-20","2026-10-21","2026-10-22","2026-10-23","2026-10-24","2026-10-25","2026-10-26","2026-10-27","2026-10-28","2026-10-29","2026-10-30","2026-10-31","2026-11-01","2026-11-02","2026-11-03","2026-11-04","2026-11-09","2026-11-10","2026-11-11","2026-11-12","2026-11-13","2026-11-14","2026-11-15","2026-11-16"],"stars":[62339,62336,62330,62320,62313,62307,62298,62295,62290,62286,62284,62280,62274,62268,62261,62253,62254,62245,62245,62242,62237,62232,62222,62215,62214,62208,62204,62195,62186,62182,62176,62167,62160,62151,62144,62143,62134,62130,62128,62122,62116,62110,62108,62103,62096,62088,62088,62083,62073,62070,62063,62060,62051,62047,62042,62043,62038,62027,62025,62021,62012,62012,61999,61994,61996,61998,61994,61994],"forks":[1686,1686,1686,1686,1685,1685,1684,1684,1685,1685,1685,1686,1687,1687,1685,1685,1686,1685,1685,1682,1681,1682,1682,1682,1682,1683,1683,1683,1684,1684,1684,1681,1681,1681,1681,1680,1681,1680,1681,1683,1683,1683,1680,1678,1677,1679,1682,1680,1680,1680,1682,1684,1690,1691,1696,1701,1701,1705,1709,1714,1708,1707,1707,1707,1707,1707,1707,1706],"watchers":[120,121,121,121,121,121,121,121,121,121,121,121,121,121,120,120,120,120,120,120,120,120,120,120,120,120,120,120,120,120,120,120,120,121,121,121,121,121,121,121,121,121,121,121,121,121,121,121,121,116,116,116,116,116,115,115,115,115,115,115,115,115,115,115,115,115,115,115],"issues_open":[null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,30,30,29,30,30,30],"prs_open":[null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,7,7,7,7,7,7]},"boards":{},"same_owner":[{"full_name":"pathwaycom/serviette","name":"serviette","shard":"1","boards":{}}],"issues":[{"number":67,"title":"Allow setting query transformers in the BaseRAGQA","comments":15,"created_at":"2024-07-22T07:41:13Z"},{"number":21,"title":"[Bug]: Simple processing of 1M rows takes more than 10 minutes","comments":12,"created_at":"2024-03-18T02:27:38Z"},{"number":17,"title":"[QUESTION] How to create accumulated batch for processing?","comments":8,"created_at":"2024-03-16T02:50:28Z"},{"number":194,"title":"[Proposal] Example: streaming RAG diagnostics with the WFGY 16-problem map in Pathway","comments":5,"created_at":"2026-02-20T13:08:41Z"},{"number":19,"title":"Add syntax to be able to reduce over multiple windows","comments":3,"created_at":"2024-03-17T20:32:26Z"},{"number":230,"title":"Entra Authentication Support (credential handler and/or password callback)","comments":2,"created_at":"2026-05-09T14:02:35Z"},{"number":75,"title":"Enable Link-Time Optimization (LTO) for Release builds","comments":2,"created_at":"2024-11-30T10:55:15Z"},{"number":254,"title":"Feature Request: SuperCompress context compression for real-time AI pipelines","comments":1,"created_at":"2026-06-30T06:00:12Z"}],"discussions":[],"latest_release":{"name":"v0.32.1","tag":"v0.32.1","published_at":"2026-08-01T09:17:35Z","body":"### Added\r\n- `pw.io.chroma.write` writes a Pathway table to a [Chroma](https://www.trychroma.com/) collection, keeping the collection in sync with the table as rows are added, changed, and removed. The columns are mapped onto Chroma's record fields explicitly: the optional `primary_key` becomes the record id (when omitted, the row's internal Pathway key is used instead), `embedding` the vector, the optional `document` column the stored text, and `metadata_columns` the record metadata. The collection must already exist. The server is addressed with `host`/`port` (plus `ssl`, `headers`, `tenant`, and `database` for authenticated deployments such as Chroma Cloud).\r\n- `pw.io.qdrant.write` writes a Pathway table to a [Qdrant](https://qdrant.tech/) collection. Each row addition is upserted as a point and each deletion removes the corresponding point, so an update replaces a point rather than duplicating it. The pre-created collection's schema drives the writing: every named vector slot of the collection is bound to the table column with the same name — dense slots to `list[float]` (or 1-D `numpy.ndarray`) columns, sparse slots to `list[tuple[int, float]]` columns of `(index, weight)` pairs — and all vectors of a point are written atomically in one upsert, enabling native hybrid (dense + BM25) search. The remaining columns are stored in the point payload. The connector fails fast on a missing collection or any slot/column mismatch instead of degrading silently. The `batch_size` parameter bounds how many points are sent per request, and an optional `api_key` authenticates against Qdrant Cloud or a secured instance.\r\n- `pw.io.duckdb.write` writes a Pathway table into a DuckDB database file through a native, in-process connector, in either `\"stream_of_changes\"` or `\"snapshot\"` mode. Embeddings stored as `numpy` arrays or `list[float]` columns land in native `DOUBLE[]` list columns, so the result is directly searchable with DuckDB's vector-distance functions for RAG retrieval. The `detach_between_batches` option makes the writer close the database after every minibatch commit and reopen it for the next one, releasing the file lock in between — so a separate process (e.g. a query server) can read committed data with short-lived read-only connections while the pipeline keeps running; the writer retries the reopen with a backoff when a reader momentarily holds the lock.\r\n- `pw.io.weaviate.write` writes a Pathway table to a [Weaviate](https://weaviate.io/) collection, keeping it in sync with the table: additions and updates upsert objects (keyed by a UUID derived from the required `primary_key`), and deletions remove them. An optional `vector` column is stored as the object's embedding; the remaining columns become object properties. The target collection must already exist. The connector writes in parallel across workers (`pathway spawn -n`), with `batch_size` and `concurrency` to tune throughput, and supports `api_key`/`headers` authentication for self-hosted and Weaviate Cloud deployments.\r\n- `pw.io.pinecone.write` writes a Pathway table to a [Pinecone](https://www.pinecone.io/) index for use as a vector store in RAG pipelines. It keeps the index in sync with the current state of the table: a row is upserted under its record id and a row removed from the table is deleted from the index. By default the id is the table's internal row key (unique, and written in parallel across workers); pass a `primary_key` column to use your own ids instead, in which case the values must uniquely identify rows (a collision raises an error) and writing runs on a single worker. The `vector` column holds either a dense embedding (`list[float]` or a 1-D `numpy.ndarray`), written to a dense index, or a sparse vector (`list[tuple[int, float]]` of `(index, weight)` pairs), written to a sparse index — so a hybrid (dense + BM25) setup is two indexes fed by two `write()` calls off the same table, with the same record ids landing in both for client-side fusion. The r","is_prerelease":false},"readme":true,"readme_zh":false,"news":[],"stale":false,"snapshot_at":"2026-11-16T09:23:14Z","history_start":"2026-09-06"}