No Vendor lock and Cloud Agnostic
Tes, switching from cloud to cloud as a data engineer – is non trivial, as a data engineer I want to learn one tool, that connects to everything i need.
Community
As a data engineer, i want to contribute from my experience to the airflow community by writing custom operators. using operator from the community is also super useful.
Engineering flexibility
Easy to Scale, easy to get the performance you want of the ETL, no missing features, as everything’s be coded easily via python. Above all , you can generate highly complicated dependencies between each step in the ETL logic.
Efficient workflows
Imagine a cutting data department – the gap between data engineering and data science is huge. The only way to bridge that gap nowadays is via SQL and Python. Airflow as a Python based ETL tool is a natural choice. An Airflow cluster based on Dusk cluster can be used for both a DE and DS.