What is Airflow?
Airflow is an open source application that gives you the ability to create and orchestrate your data pipelines.
- Complex data pipeline can be built using airflow with complex dependencies, retries mechanism and triggering rules.
- Cloud vendor Agnostic, i.e works well with all clouds.
Airflow use cases:
Batch processing data pipelines.
Airflow antipattern use cases:
Real time data pipelines.
Our Airflow Blogs
Airflow use cases
- Airflow guidelines and data pipelines best practices
- Google Analytics Full ETL
- Similarweb API data pipeline via Airflow
- DFP Data Transfer Files Use Case | Airflow and BigQuery 93% Cost Reduction
- How to connect GCP Composer cloud SQL database ? and how to migrate data to BQ?
- How to convert colabs notebook into python and scheduling it in Apache Airflow?
- Airflow Use Case: Improving success rate of API calls
- AppsFlyer Data Locker Airflow pipeline use case
- Google Search Console API via Airflow
- Airflow Dynamic Operators used in Google Ad Manager API data pipeline
- How to connect to Google Ad Manager via Airflow
- Connecting to Facebook API via Airflow
- Why Choose Airflow and why not?
- Airflow Exception: “raise InvalidToken cryptography.fernet.InvalidToken”
- Airflow file sensor example
- Airflow examples of jobs
- Airflow AttributeError: ‘NoneType’ object has no attribute ‘create_dagrun’
- How to ssh to a remote GCP machine and run a command via Airflow ?
- How to debug BigQuery query failure in Airflow DAG?
- Why the author of “We’re All Using Airflow Wrong and How to Fix It” needs to attend my meetup!
- Airflow XComs example
Airflow setup - "how to"
- Airflow – distinguish between environments
- Airflow SequentialExecutor Installation Centos 7.6
- Airflow SequentialExecutor Installation manual and basic commands
- Everything you need to know about installing a DIY LocalExecutor Airflow cluster backed by MySQL Cloud SQL
- Airflow Performance tuning in 5 min
- mounting GCS as FUSE for Airflow
- Airflow – setup of SSL Certificate – HTTPS example
- Airflow MySQL Integration – how to?
- Airflow and Slack Integration
- How to Setup SMTP for Airflow