How to connect GCP Composer cloud SQL database ? and how to migrate data to BQ?

Author: Omid Vahdaty 16.10.2019

In this blog I will connect to Airflow Cloud SQL Database which is in the Tenant Project of the Composer Environment [1]

You can do that by installing the SQLAlchemy toolkit and following these instructions [2].
Or you could also remotely authorize access to your Cloud SQL database from an application, client, or other GCP service, Cloud Composer provides the Cloud SQL proxy in the GKE cluster [3]


How to migrate data from GCP Composer to BigQuery?

After researching your question, I found this documentation link[1] which tells you how to access the Cloud SQL database of Composer from another product, and this other one[2], which explains how to integrate BigQuery and Cloud SQL.


BigQuery and Federated queries of the composer backed database Cloud SQL?

Using BigQuery federated queries to access the Cloud SQL instance that runs within the GKE cluster of a Composer environment is not possible. The only way to access the SQL instance data is using a Cloud SQL proxy from the GKE cluster

I put a lot of thoughts into these blogs, so I could share the information in a clear and useful way.
If you have any comments, thoughts, questions, or you need someone to consult with,,

feel free to contact me via LinkedIn:

1 thought on “How to connect GCP Composer cloud SQL database ? and how to migrate data to BQ?”

Leave a Reply