GitHub Store
apache

airflow

apache

Apache Airflow is an open source platform to programmatically author, schedule, and monitor workflows, defining task dependencies as Python DAGs for ETL, machine learning, and other data pipelines.

3.3.2
Published at
Sep 17, 2026
47,223
Stars
2,997
Download
Synced at
8d ago

Latest release

3.3.2Sep 17, 2026, 11:10 AM
Downloads slow? Switch the source anytime above

README

Project Overview

Apache Airflow is a platform to programmatically author, schedule, and monitor workflows. Workflows (DAGs) are defined as Python code, which makes them more maintainable, versionable, testable, and collaborative; the scheduler executes your tasks on an array of workers following the specified dependencies, while rich command line utilities and a rich user interface make it easy to visualize production pipelines, monitor progress, and troubleshoot issues.

Item Value
Platform Python environment, Docker, and more
Language Python
License Apache-2.0
Core concepts DAGs, task dependencies, scheduling and monitoring

Workflows as code: define once, with scheduling, monitoring, and collaboration built in.

Key Features

  • Workflows as code: DAGs are written in Python, so version control and code review come naturally
  • Task scheduling: the scheduler runs tasks on multiple workers according to dependencies and timing
  • Visual monitoring: the web UI shows pipeline status, progress, and failure details
  • Command line tools: rich CLI utilities perform complex operations on DAGs
  • Broad scenarios: orchestrates ETL, ELT, machine learning, and MLOps workflows
  • Container deployment: an official Docker image eases running on various infrastructures

Target Audience

  • Data engineers
  • Data platform teams
  • MLOps engineers
  • Anyone scheduling batch jobs

Quick Start

  1. Install from PyPI:
pip install apache-airflow
  1. Start a local development environment in one command:
airflow standalone
  1. Open the web UI and explore the bundled example DAGs
  2. Follow the official tutorial to write your first DAG

Core Advantages

  • Collaborative by design: workflows live as code, ready for review, rollback, and testing
  • Intuitive UI: pipeline health is visible at a glance, lowering debugging cost
  • Mature ecosystem: an Apache Software Foundation project with a large community and many integrations

System Requirements

Item Requirement
Runtime Python or Docker
Deployment PyPI install or the official Docker image

Notes

  • Learning curve: the DAG model and scheduling semantics take some study; finish the official tutorial before going live
  • Production deployment: airflow standalone is for local experience only; plan the metadata database, workers, and other components yourself in production

Release history

Sort