Apache Airflow

We previously used Apache Airflow for orchestrating ML workflows, but are moving away from it in favor of more lightweight solutions like Celery for Python services.

Airflow is now on hold - we don't recommend starting new projects with it and are migrating existing workflows to alternative solutions that better fit our infrastructure and operational requirements.

https://airflow.apache.org/

Apache Airflow is a workflow orchestration platform for scheduling and monitoring data pipelines.

We use Apache Airflow specifically for orchestrating machine learning workflows. This is a limited-scope implementation focused on ML pipeline management.

📚 Resources

• Apache Airflow
• Airflow Documentation