How to ssh to a remote GCP machine and run a command via Airflow ?
Author: Omid Vahdaty 4.11.2019
Use an Airflow Operator- BashOperator which allows to execute bash commands locally in the node that are running the Airflow Workers. You can use the gcloud command to connect to a remote instance. Below is an example, also committed in our git.
import datetime import os import logging from airflow import models from airflow.contrib.operators import bigquery_to_gcs from airflow.contrib.operators import gcs_to_bq from airflow.operators.dummy_operator import DummyOperator from airflow.operators import BashOperator from airflow.contrib.operators import gcs_to_gcs #from airflow.utils import trigger_rule yesterday = datetime.datetime.combine( datetime.datetime.today() - datetime.timedelta(1), datetime.datetime.min.time()) default_dag_args = { # Setting start date as yesterday starts the DAG immediately when it is # detected in the Cloud Storage bucket. 'start_date': yesterday, # To email on failure or retry set 'email' arg to your email and enable # emailing here. 'email_on_failure': False, 'email_on_retry': False, # If a task fails, retry it once after waiting at least 5 minutes 'retries': 1, 'retry_delay': datetime.timedelta(minutes=5), 'project_id': models.Variable.get('gcp_project') } bash_cmd='gcloud beta compute --project MyProjectName ssh myMachineHostname --internal-ip --zone us-central1-a --command "ls /tmp/"' with models.DAG( 'bash_remote_gcp_machine_example', # Continue to run DAG once per day schedule_interval="@once", default_args=default_dag_args) as dag: start = DummyOperator(task_id='start') end = DummyOperator(task_id='end') bash_remote_gcp_machine = BashOperator(task_id='bash_remote_gcp_machine_task',bash_command=bash_cmd) start >> bash_remote_gcp_machine >> end
The above airflow will only work if the service account is allowed to access the remote machine. If not, use the below authentication command combined BashOperator:
#Authenticating with a Service Account key bash_command_auth = BashOperator(task_id='auth_bash',bash_command = 'gcloud auth activate-service-account --key-file=//<your_service_account_key.json')
——————————————————————————————————————————
I put a lot of thoughts into these blogs, so I could share the information in a clear and useful way.
If you have any comments, thoughts, questions, or you need someone to consult with,
feel free to contact me via LinkedIn: