When to use BigQuery? When not to use BigQuery?

When to use BigQuery?

1. 80% of all data use case would be a good reason to use BigQuery.
2. If you are not sure – start with BigQuery
3. If you are using google analytics or firebase
4. When you have a director connector that load the data for you to BigQuery
5. When your production is in GCP

When not to use BigQuery [Notice the below are the other 20% or less – Extreme cases which are not common]:

1. When are processing more than one 5GB compressed file per hour (load limit of GCP BigQuery per file). Try opening a compressed file of 6GB and splitted into smaller files… your one hour window of time will be shining as the file uncompressed into something huge of ±20 GB ….
2. When you are hitting an error in BigQuery that says your query is consuming too much resources, ona weekly basis, and there nothing you can do about that.
3. When you need to self Join Billion of records on a regular basis.
4. When you you are using complex Window functions. You are likely to get an error of too many resource are being user for your Query and there nothing you can do except rewriting your query.

So what are the alternative to BigQuery:

1. Hadoop ecosystem: Data Proc / Cloudera
2. SQream DB , a database designed to handle huge files, massive joins at surprising amount of speed, simplicity and cost effectiveness.