Top 5 Reasons to choose Apache Airflow

No Vendor lock and Cloud Agnostic

Tes, switching from cloud to cloud as a data engineer – is non trivial, as a data engineer I want to learn one tool, that connects to everything i need.

Community

As a data engineer, i want to contribute from my experience to the airflow community by writing custom operators. using operator from the community is also super useful.

Engineering flexibility

Easy to Scale, easy to get the performance you want of the ETL, no missing features, as everything’s be coded easily via python. Above all , you can generate highly complicated dependencies between each step in the ETL logic.

Efficient workflows

Imagine a cutting data department – the gap between data engineering and data science is huge. The only way to bridge that gap nowadays is via SQL and Python. Airflow as a Python based ETL tool is a natural choice. An Airflow cluster based on Dusk cluster can be used for both a DE and DS.

Top 3 Reasons Not to Choose Airflow

Learning Curve

Sometimes, your organization can not afford letting you do the learning curve on company time. so there is readly no time to implement the tool properly. Airflow is more useful when you have a team of data engineers, and the amount of ETL is growing month over month, However, most SMB are not scaling their data teams that quickly so Airflow may be an overkill.

Time to market

Writing an ETL via airflow sometimes takes a bit longer that modern tools.

Debug is a challenge

Debugging an ETL via airflow is a a bit more challenging as errors are not clear.

Not everything is supported out of the box

Airflow connects well in GCP ecosystem. I.e there is an Airflow Operator for all the basics services in GCP . However in AWS the list of Airflow operator is shorter, you may need to rely on AWS Boto3 for the basics.