mounting GCS as FUSE for Airflow

Full instructions can be found here, However this blog to make things simpler 🙂

• Add the gcsfuse distribution URL as a package source and import its public key:

export GCSFUSE_REPO=gcsfuse-`lsb_release -c -s`
echo "deb http://packages.cloud.google.com/apt $GCSFUSE_REPO main" | sudo tee /etc/apt/sources.list.d/gcsfuse.list
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -

• Update the list of packages available and install gcsfuse.

sudo apt-get update 
sudo apt-get install -y gcsfuse 

• Create a directory if not already created on setup of airflow:

mkdir ~/airflow/dags

• create target bucket in GCS (using the GUI is ok ) , e.g bucket name: gs://airflow-fuse-bucket

• Use Cloud Storage FUSE to mount the bucket :

gcsfuse airflow-fuse-bucket ~/airflow/dags
ls ~/airflow/dags

• don’t forget to add the mounts to your automation of starting airflow
• dont forget also to change the Airflow.cfg to the correct path for logs and dags

——————————————————————————————————————————

I put a lot of thoughts into these blogs, so I could share the information in a clear and useful way. If you have any comments, thoughts, questions, or you need someone to consult with,