Uncategorized

bigquery bq load error- “cannot determine table described”

25th April 20215th May 2021 Omid

BigQuery bq load error- "cannot determine table described"

Author: Omid Vahdaty 21.4.2021

If you are are getting this error , it is an authentication and authorization issue, simply log out and log in again. e.g if you are using cloud shell – close it and reopen.

Commands like the below will describe your project and data set – but still wont sent the cmd to BQ API:

bq show mydataset.my_test
bq show mydataset

You can also try adding explicit project id as follows:

projectid:myset.my_test

——————————————————————————————————————————
I put a lot of thoughts into these blogs, so I could share the information in a clear and useful way.
If you have any comments, thoughts, questions, or you need someone to consult with,

airflow

Airflow Exception: “raise InvalidToken cryptography.fernet.InvalidToken”

18th April 202118th April 2021 Omid

Airflow Exception: "raise InvalidToken cryptography.fernet.InvalidToken"

Author: Omid Vahdaty 5.4.2021

If you get this error of invalid token, it is because Airflow is using Fernet. Airflow encrypt all the passwords for its connections in the backend database.

Somehow Airflow backend is using previous fernet key and you have generated a key which you have created in a new connection.

My recommendation is to do the following first:

This will help in deleting all the existing records in your backend db. NOTICE – this will delete all your Airflow connections and variables you inputted manually:

airflow resetdb

airflow initdb

This will initialize backend db like a fresh install db. Airflow may shout about missing variables.
Start Airflow and enter missing variables one by one.

Then start airflow web server and scheduler.

feel free to contact me via LinkedIn:

Data Analysis from raw data to dashboarding

7th February 20217th February 2021 Omid

Data Analysis from raw data to dashboarding

Lecturer: David Zenesh 2.2.2021

In the world of big data, data visualization tools and technologies are essential to analyze massive amounts of information and make data-driven decisions. Today’s culture is visual, including everything from art and advertisements to TV and movies, and our eyes are drawn to colors and patterns. Our interaction with data should reflect this reality.

Video

feel free to contact me via LinkedIn – Omid Vahdaty:

NoSQL

Extreme Data Streaming using Reactive and Aerospike NoSQL

10th January 202111th January 2021 Omid

Extreme Data Streaming using Reactive and Aerospike NoSQL

Lecturer: Rony Keren 6.1.2020

How Reactive Programming allows implementing massive data streaming and event pooling by using optimal, limited number of threads. Combined with rapid disk access and IO, it reduces the usage of expensive cache, simplify and speedup access to data, mostly relevant for Big Data.

Video and slides

Next-Generation Real-time NoSQL database

Lecturer: Zohar Elkayam 6.1.2020

How Aerospike distributed NoSQL Database, with its patented Hybrid Memory Architecture TM can be used for building sub-millisecond high throughput real-time database applications to handle any data at any scale, in the fastest and easiest way and in the lowest possible costs.

Video and slides

feel free to contact me via LinkedIn – Omid Vahdaty:

How to improve Tableau Dashboard Performance

6th January 20217th January 2021 Omid

How to improve Tableau dashboard performance

5 Tips on how to optimize your dashboard performance in Tableau

Author: Delphine Elfassi 6.1.2021

Avoid calculated fields & custom queries.
Extract your data
Add filters to context.
Avoid quick filters.
Customize your filters.

1 | Avoid calculated fields and Custom Queries

If possible, try to avoid calculated fields and custom queries – You should perform calculations in your database.

2 | Extract Your Data

Use the Extract connection instead of the live connection to your data source.
For More Details : https://help.tableau.com/current/pro/desktop/en-us/extracting_data.htm

3 | Choose “Add to Context” to your Filters

By default, all filters are calculated independently. That means that for each filter Tableau runs through the entire data source and figures out all the rows that pass each filter individually and then returns the union of the results. For example, say you’ve filtered a view to only show the Western region and you’ve added another filter to only show sales from the first quarter of 2009. Tableau first looks at all the records and pulls out all the ones from the Western region. Then it goes back through all of the records again and pulls out all sales from Q1 of 2009. Finally the result is the union of both these independent filters.

Sometimes you’ll want to first run one filter and then run other filters just on the results of the first one. These types of filters are called Context Filters. When a filter is added to the context it is the independent filter and all other filters are only computed just on the results of the context.

NOTE: If your table has complex aggregations and is more than a simple view table, you should be very careful using this option!

For More Details :
https://help.tableau.com/current/pro/desktop/en-us/filtering_context.htm

Go to Filters Area → Right Click on the filter field → Add to Context.

4 | Avoid Quick Filters

Use hierarchical / Cascading quick filters only If your filters do not contain too many values.
The disadvantage of quick filters is that they will increase the loading time, that’s why it is recommended to use only necessary quick filters and do not add more quick filters in your worksheet or Dashboard.

Go to the show filter → Right Click → All values in Database

5 | Customize your filter

Customize your filter and choose the “Apply” button.

Right click on the filter show → Customize → “Show Apply Button”

feel free to contact me via LinkedIn: