mounting GCS as FUSE for Airflow

Full instructions can be found here, However this blog to make things simpler 🙂

  • Add the gcsfuse distribution URL as a package source and import its public key:
export GCSFUSE_REPO=gcsfuse-`lsb_release -c -s`
echo "deb http://packages.cloud.google.com/apt $GCSFUSE_REPO main" | sudo tee /etc/apt/sources.list.d/gcsfuse.list
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
  • Update the list of packages available and install gcsfuse.
sudo apt-get update 
sudo apt-get install -y gcsfuse 
  • Create a directory if not already created on setup of airflow:
mkdir ~/airflow/dags
  • create target bucket in GCS (using the GUI is ok ) , e.g bucket name: gs://airflow-fuse-bucket
  • Use Cloud Storage FUSE to mount the bucket :
gcsfuse airflow-fuse-bucket ~/airflow/dags
ls ~/airflow/dags
  • don’t forget to add the mounts to your automation of starting airflow
  • dont forget also to change the Airflow.cfg to the correct path for logs and dags

——————————————————————————————————————————

I put a lot of thoughts into these blogs, so I could share the information in a clear and useful way. If you have any comments, thoughts, questions, or you need someone to consult with, 

feel free to contact me via LinkedIn: