Tag: Big Query

Uncategorized

Keep your data encrypted in BigQuery

Omid

Keep your data encrypted in BigQuery

Lecturer: Ran Tibi 6.2.2022

If you work with data and build a data warehouse you probably have some sensitive data that you want to keep secured.
While BigQuery encrypts all the data before it is written to the disk, once you have read access to the tables you can have full visibility of the data, including sensitive data and PII.
In this talk, we will describe a use case that shows how you can create a secure end-to-end process that encrypts at the application level the data before inserting it into BigQuery and allow users to decrypt it only in query time without the need of knowing the actual encryption key.

Video

Slides


——————————————————————————————————————————
I put a lot of thoughts into these blogs, so I could share the information in a clear and useful way.
If you have any comments, thoughts, questions, or you need someone to consult with,

feel free to contact me via LinkedIn – Omid Vahdaty:

Linkedin
Uncategorised

BigQuery Demystified

Omid

BigQuery Demystified

Lecturer: Omid Vahdaty 12.7.2022

In this session, We are going to review the basics of BigQuery SQL and practice with live demo.

Video

Slides

How to create a table in BigQuery with Partitioning and clustering via DDL command?

The following code snippet is the answer, however, be very careful about what you choose as cluster fields as this may cause unintended performance and costs penalties.

				
					CREATE TABLE my_dataset.new_clustered_table 
( 
date DATE, 
field_a STRING, 
field_b STRING, 
field_c STRING, 
field_d STRING, 
field_e STRING ) 
PARTITION BY date 
CLUSTER BY field_a, field_b 
OPTIONS ( description="a_table clustered_by_two_fields" )

This BigQuery create table example is also committed in our github

some relevant docs:

[1]: https://cloud.google.com/bigquery/docs/reference/standard-sql/data-definition-language#creating_a_clustered_table_from_the_result_of_a_query 

[2]: https://cloud.google.com/bigquery/docs/clustered-tables

 [3]: https://cloud.google.com/bigquery/docs/creating-clustered-tables#limitations

How to do something like show create table in BigQuery?

The short answer: You can’t. The good news: There is a work around.

				
					SELECT * FROM `myProjecId1234.MyDataSet.Table_Schema` 

where table_schema ='MyDataSet' and table_name ='MyTable'

The example is also committed in our GitHub

A good read about BigQuery implementation in Twitter:

https://blog.twitter.com/engineering/en_us/topics/infrastructure/2019/democratizing-data-analysis-with-google-bigquery.html


——————————————————————————————————————————
I put a lot of thoughts into these blogs, so I could share the information in a clear and useful way.
If you have any comments, thoughts, questions, or you need someone to consult with,

feel free to contact me via LinkedIn – Omid Vahdaty:

Linkedin
For more BigQuery blogs
architecture, GCP Big Data Demystified, superQuery

80% Cost Reduction in Google Cloud BigQuery

Omid

BigQuery Cost Reduction Demystified

Lecturer: Omid Vahdaty 15.6.2022

In this lecture I will share with how I saved 80% of BigQuery monthly billing of investing.com.
How to reduce costs using GCP big Query? what should we pay attention to?
We are going to cover all of google best practices while working with BigQuery.

Video

Slides

27.10.2019

Video


——————————————————————————————————————————
I put a lot of thoughts into these blogs, so I could share the information in a clear and useful way.
If you have any comments, thoughts, questions, or you need someone to consult with,

feel free to contact me via LinkedIn – Omid Vahdaty:

Linkedin
For more GCP BigQuery blogs