Airflow Performance tuning in 5 min
Author: Omid Vahdaty 23.3.2019
When you are using dynamic operators , the default settings will not work.I suggest the below settings.
Notice this configuration is provided AS IS.
Usually the config file is located in ~/airflow/airflow.cfg
This was tested in DEV environment only. Be sure to understand what your are doing.
# in the pool. 0 indicates no limit. default is 5 sql_alchemy_pool_size = 0 # max_overflow can be set to -1 to indicate no overflow limit; # no limit will be placed on the total number of concurrent connections. sql_alchemy_max_overflow = -1 # the max number of task instances that should run simultaneously # on this airflow installation parallelism = 64 # The number of task instances allowed to run concurrently by the scheduler # i suggest double the defaults after installation. dag_concurrency = 32 # The maximum number of active DAG runs per DAG max_active_runs_per_dag = 32 # How long before timing out a python file import # default is 30, i suggest 3000 for dynamic dag operators. dagbag_import_timeout = 3000 # How long before timing out a DagFileProcessor, which processes a dag file # default is 50 , i suggest 5000 dag_file_processor_timeout = 5000 # The scheduler can run multiple threads in parallel to schedule dags. # This defines how many threads will run. my default was 2. i suggest 4 times the #default. max_threads = 8 #consider change the heart rate scheduling of scheduler If the last scheduler heartbeat happened more than scheduler_health_check_threshold # ago (in seconds), scheduler is considered unhealthy. # This is used by the health check in the "/health" endpoint scheduler_health_check_threshold = 60 # scheduler_zombie_task_threshold should be higher then # scheduler_zombie_task_threshold scheduler_zombie_task_threshold = 120
I put a lot of thoughts into these blogs, so I could share the information in a clear and useful way.
If you have any comments, thoughts, questions, or you need someone to consult with,
feel free to contact me via LinkedIn: