Airflow

What is Airflow?

Airflow is an open source application that gives you the ability to create and orchestrate your data pipelines.

Airflow advantages:

  1. Complex data pipeline can be built using airflow with complex dependencies, retries mechanism and triggering rules.
  2. Cloud vendor Agnostic, i.e works well with all clouds. 

Airflow use cases:

Batch processing data pipelines.

Airflow antipattern use cases:

Real time data pipelines.

Our Airflow Blogs

Airflow use cases

  1. Airflow guidelines and data pipelines best practices
  2. Google Analytics Full ETL
  3. Similarweb API data pipeline via Airflow
  4. DFP Data Transfer Files Use Case | Airflow and BigQuery 93% Cost Reduction
  5. How to connect GCP Composer cloud SQL database ? and how to migrate data to BQ? 
  6. How to convert colabs notebook into python and scheduling it in Apache Airflow?
  7. Airflow Use Case: Improving success rate of API calls
  8. AppsFlyer Data Locker Airflow pipeline use case 
  9. Google Search Console API via Airflow
  10. Airflow Dynamic Operators used in Google Ad Manager API data pipeline 
  11. How to connect to Google Ad Manager via Airflow
  12. Connecting to Facebook API via Airflow
  13. Why Choose Airflow and why not?

Airflow examples

  1. Airflow Exception: “raise InvalidToken cryptography.fernet.InvalidToken”
  2. Airflow file sensor example 
  3. Airflow examples of jobs
  4. Airflow AttributeError: ‘NoneType’ object has no attribute ‘create_dagrun’
  5. How to ssh to a remote GCP machine and run a command via Airflow ?
  6. How to debug BigQuery query failure in Airflow DAG?
  7. Why the author of “We’re All Using Airflow Wrong and How to Fix It” needs to attend my meetup!
  8. Airflow XComs example

Airflow setup - "how to"

  1. Airflow – distinguish between environments
  2. Airflow SequentialExecutor Installation Centos 7.6
  3. Airflow SequentialExecutor Installation manual and basic commands
  4. Everything you need to know about installing a DIY LocalExecutor Airflow cluster backed by MySQL Cloud SQL
  5. Airflow Performance tuning in 5 min
  6. mounting GCS as FUSE for Airflow
  7. Airflow – setup of SSL Certificate – HTTPS example
  8. Airflow MySQL Integration – how to?
  9. Airflow and Slack Integration
  10. How to Setup SMTP for Airflow

Architectures and meetups which includes Airflow

English

  1. Airflow Performance and Best Practices Demystified
  2. Airflow Demystified
  3. DE Use Cases: Airflow, Cost reduction,BQ ML

Hebrew

BigQuery VS Dataproc cost reduction use case

Top Videos English