pathway

Project Overview

Pathway is a Python ETL framework for real-time data processing: one Python codebase runs batch jobs, consumes continuous streams such as Kafka, and builds LLM and RAG pipelines. Though the API is Python, execution is driven by a scalable Rust engine based on Differential Dataflow that performs incremental computation with multithreading, multiprocessing, and distributed deployment; pipelines stay fully in memory and deploy easily with Docker and Kubernetes. The same code serves local development, CI tests, batch jobs, stream replays, and live streams.

One codebase for batch and streaming, with C++-grade performance from the Rust engine.

Key Features

• Batch and streaming unified: the same code handles static data and continuous streams, switching between them with zero changes
• Rich connectors: built-in connectors for Kafka, PostgreSQL, GDrive, SharePoint, and more; the Airbyte connector reaches 300+ sources, and custom connectors are supported
• Stateless and stateful transforms: joins, windowing, and sorting implemented natively in Rust, while any Python function can be embedded in the pipeline
• LLM and RAG tooling: wrappers for major LLM services and vector search components, so live data sources feed dynamic RAG directly
• Persistence: computation state can be saved, letting pipelines resume after updates or crashes
• Consistency: the framework manages time semantics to keep all computations consistent
• Incremental computation: updates recompute only affected parts instead of full reruns
• Scalable deployment: the Rust engine supports multithreading, multiprocessing, and distribution with in-memory performance

Target Audience

• Data engineers building real-time pipelines
• Teams developing dynamic RAG and LLM apps
• Operations scenarios needing live monitoring and alerting
• Developers handling Kafka / IoT streams

Quick Start

1. Install: pip install -U pathway
2. Define a pipeline (watching a directory):

import pathway as pw

t = pw.io.fs.read('./input', format='csv')
result = t.groupby(t.key).reduce(total=pw.reducers.sum(t.value))
pw.io.csv.write(result, './output')
pw.run()

3. For streaming, swap the source to pw.io.kafka.read and keep the rest unchanged
4. The official template library offers runnable examples for real-time ETL, log alerting, and RAG

Core Advantages

• Develop equals production: one codebase covers local dev, CI, batch jobs, and streaming — no stream rewrite needed
• Strong performance: Python on the outside, Rust on the inside, with incremental updates at minimal cost
• Simple deployment: fully in-memory pipelines with batteries-included Docker and Kubernetes support

System Requirements

Notes

• License model: BSL 1.1 — free for non-production use; review LICENSE for commercial and resale terms
• LLM features are xpack: some advanced LLM wrappers live in the xpack module; check its docs before use