Big Query, superQuery

Quick Start on GCP BigQuery – 5 rules of thumb

1. Install BQ mate – plugin for chrome in order to see cost per query in dollars before running.

2. log in to your BigQuery console to get started, consider using superQuery IDE for more advanced features over BigQuery, Increased productivity and cost reduction.

3. run sample query , ALWAYS use a date in the where clause, assuming the table is partitioned, this will save you costs by only scanning the relevant data as oppose to the entire table.

select * from 
where date = '2019-11-01'

4. AS rule of thumb, NEVER DO the below:

select * from t
or
select * from t limit 100

as this will incur unintended full scans of table data and thus create huge billing costs. Instead, use:

Select ColName1, ColName2 from T

and Preview table instead of LIMIT


5. as a rule of thumb , ALWAYS
:

if you are running a query more than 1$ – you are most probably doing something wrong.

——————————————————————————————————————————

I put a lot of thoughts into these blogs, so I could share the information in a clear and useful way. If you have any comments, thoughts, questions, or you need someone to consult with, feel free to contact me:

https://www.linkedin.com/in/omid-vahdaty/

architecture, Big Query, cost reduction, GCP Big Data Demystified, superQuery

80% Cost Reduction in Google Cloud BigQuery | Tips and Tricks | Big Query Demystified | GCP Big Data Demystified #2

The second in series of lectures GCP Big Data Demystified. In this lecture I will share with how I saved 80% of BigQuery monthly billing of investing.com. Lectures slides:

Videos from the meetup:

Link to previous lecture GCP Big Data Demystified #1

——————————————————————————————————————————

I put a lot of thoughts into these blogs, so I could share the information in a clear and useful way. If you have any comments, thoughts, questions, or you need someone to consult with, feel free to contact me:

https://www.linkedin.com/in/omid-vahdaty/

To learn more about superQuery: