How to connect GCP Composer cloud SQL database ? and how to migrate data to BQ?

Author: Omid Vahdaty 16.10.2019

In this blog I will connect to Airflow Cloud SQL Database which is in the Tenant Project of the Composer Environment [1]

You can do that by installing the SQLAlchemy toolkit and following these instructions [2].
Or you could also remotely authorize access to your Cloud SQL database from an application, client, or other GCP service, Cloud Composer provides the Cloud SQL proxy in the GKE cluster [3]

[1] https://cloud.google.com/composer/docs/concepts/overview#architecture
[2] https://cloud.google.com/composer/docs/how-to/using/installing-python-dependencies#installing_sqlalchemy_to_access_the_airflow_database
[3] https://cloud.google.com/sql/docs/mysql/connect-kubernetes-engine

How to migrate data from GCP Composer to BigQuery?

After researching your question, I found this documentation link[1] which tells you how to access the Cloud SQL database of Composer from another product, and this other one[2], which explains how to integrate BigQuery and Cloud SQL.

[1]: https://cloud.google.com/composer/docs/concepts/overview#cloud-sql
[2]: http://googlecloudsql.tobigquery.com/

BigQuery and Federated queries of the composer backed database Cloud SQL?

Using BigQuery federated queries to access the Cloud SQL instance that runs within the GKE cluster of a Composer environment is not possible. The only way to access the SQL instance data is using a Cloud SQL proxy from the GKE cluster


——————————————————————————————————————————
I put a lot of thoughts into these blogs, so I could share the information in a clear and useful way.
If you have any comments, thoughts, questions, or you need someone to consult with,,

feel free to contact me via LinkedIn:

1 thought on “How to connect GCP Composer cloud SQL database ? and how to migrate data to BQ?”

Leave a Reply

Discover more from Big Data Demystified

Subscribe now to keep reading and get access to the full archive.

Continue reading