airflow
Apache Airflow workflow orchestration. Use for data pipelines.
Works with
Claude CodeCursorCodex CLIGitHub CopilotGemini CLI
---
name: airflow
description: Apache Airflow workflow orchestration. Use for data pipelines.
license: MIT
---
# Airflow
Apache Airflow is the standard for data engineering pipelines. v3.0 (2025) introduces **Event-driven Triggers** and a modern React UI.
## When to Use
- **ETL/ELT**: Scheduling nightly data warehouse loads.
- **ML Ops**: Retraining models when new data arrives.
- **Dependency Management**: "Run Task B only if Task A succeeds".
## Core Concepts
### DAGs (Directed Acyclic Graphs)
Defined in Python.
### Task SDK
New in v3.0. Allows writing tasks in any language, not just Python.
### Edge Executor
Run tasks on remote edge devices.
## Best Practices (2025)
**Do**:
- **Use the TaskFlow API**: `@task` decorators are cleaner than `PythonOperator`.
- **Use Datasets**: Define data-aware scheduling (`schedule=[Dataset("s3://bucket/file")]`).
**Don't**:
- **Don't put top-level code in DAG files**: It runs every scheduler heartbeat.
## References
- [Airflow Documentation](https://airflow.apache.org/)More Data Engineering skills
data-pipeline
claude-office-skills/skills
Data pipeline and ETL automation - extract, transform, load workflows for data integration and analytics
4.1k
ETL Pipeline
claude-office-skills/skills
Design and automate Extract, Transform, Load data pipelines for data integration and analytics
3.9k
data-throughput-accelerator
affaan-m/ecc
Use when large data ingestion, backfill, export, ETL, warehouse loading, manifest catch-up, or table synchronization needs to become much faster while preserving data correctness.
3.6k
