How to ssh to a remote GCP machine and run a command via Airflow ?

Author: Omid Vahdaty 4.11.2019​

Use an Airflow Operator- BashOperator which allows to execute bash commands locally in the node that are running the Airflow Workers. You can use the gcloud command to connect to a remote instance. Below is an example, also committed in our  git. 

import datetime
import os
import logging
from airflow import models
from airflow.contrib.operators import bigquery_to_gcs
from airflow.contrib.operators import gcs_to_bq
from airflow.operators.dummy_operator import DummyOperator
from airflow.operators import BashOperator
from airflow.contrib.operators import gcs_to_gcs
#from airflow.utils import trigger_rule
yesterday = datetime.datetime.combine(
    datetime.datetime.today() - datetime.timedelta(1),
    datetime.datetime.min.time())
default_dag_args = {
    # Setting start date as yesterday starts the DAG immediately when it is
    # detected in the Cloud Storage bucket.
    'start_date': yesterday,
    # To email on failure or retry set 'email' arg to your email and enable
    # emailing here.
    'email_on_failure': False,
    'email_on_retry': False,
    # If a task fails, retry it once after waiting at least 5 minutes
    'retries': 1,
    'retry_delay': datetime.timedelta(minutes=5),
    'project_id': models.Variable.get('gcp_project')
}
bash_cmd='gcloud beta compute --project MyProjectName ssh myMachineHostname --internal-ip --zone us-central1-a --command "ls /tmp/"'
with models.DAG(
        'bash_remote_gcp_machine_example',
        # Continue to run DAG once per day
        schedule_interval="@once",
        default_args=default_dag_args) as dag:
     start = DummyOperator(task_id='start')
    
     end = DummyOperator(task_id='end')
         
     bash_remote_gcp_machine = BashOperator(task_id='bash_remote_gcp_machine_task',bash_command=bash_cmd)
	
start >> bash_remote_gcp_machine >> end

The above airflow will only work if the service account is allowed to access the remote machine. If not, use the below authentication command combined BashOperator:

#Authenticating with a Service Account key
bash_command_auth = BashOperator(task_id='auth_bash',bash_command = 'gcloud auth activate-service-account --key-file=//<your_service_account_key.json')

——————————————————————————————————————————
I put a lot of thoughts into these blogs, so I could share the information in a clear and useful way.
If you have any comments, thoughts, questions, or you need someone to consult with,

feel free to contact me via LinkedIn:

Leave a Reply