Luigi
Python-based workflow orchestration framework from Spotify for building complex data pipelines with dependency resolution.
✨ Key Features
- ✓ Python tasks: Define work units as classes
- ✓ Dependencies: Automatic resolution and ordering
- ✓ Targets: Output tracking (files, databases)
- ✓ Parameters: Configurable task inputs
- ✓ Hadoop: HDFS and MapReduce support
- ✓ Spark: PySpark task support
- ✓ Databases: PostgreSQL, MySQL, Redshift
- ✓ Cloud: S3, GCS file targets
- ✓ Central scheduler: Coordinates task execution
- ✓ Worker processes: Parallel execution
- ✓ Retries: Configurable failure handling
- ✓ Idempotency: Target-based completion tracking
- ✓ Dependency graph: Visual task relationships
- ✓ Task history: Execution timeline
- ✓ Web interface: Basic monitoring UI
📁 More Orchestration Tools
Apache Airflow
The industry-standard workflow orchestration platform
Argo Workflows
Dagster
Cloud-native orchestration platform for data pipelines
dbt Cloud