mounting GCS as FUSE for Airflow
Author: Omid Vahdaty 15.10.2019
Full instructions can be found here, However this blog to make things simpler 🙂
- Add the gcsfuse distribution URL as a package source and import its public key:
export GCSFUSE_REPO=gcsfuse-`lsb_release -c -s` echo "deb http://packages.cloud.google.com/apt $GCSFUSE_REPO main" | sudo tee /etc/apt/sources.list.d/gcsfuse.list curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
- Update the list of packages available and install gcsfuse.
sudo apt-get update sudo apt-get install -y gcsfuse
- Create a directory if not already created on setup of airflow:
mkdir ~/airflow/dags
- create target bucket in GCS (using the GUI is ok ) , e.g bucket name: gs://airflow-fuse-bucket
- Use Cloud Storage FUSE to mount the bucket :
gcsfuse airflow-fuse-bucket ~/airflow/dags ls ~/airflow/dags
- don’t forget to add the mounts to your automation of starting airflow
- dont forget also to change the Airflow.cfg to the correct path for logs and dags
——————————————————————————————————————————
I put a lot of thoughts into these blogs, so I could share the information in a clear and useful way. If you have any comments, thoughts, questions, or you need someone to consult with,Â
feel free to contact me via LinkedIn:
1 thought on “mounting GCS as FUSE for Airflow”