Blog

architecture, Big Query, cost reduction, GCP Big Data Demystified, superQuery

80% Cost Reduction in Google Cloud BigQuery

BigQuery Cost Reduction Demystified

Lecturer: Omid Vahdaty 15.6.2022

In this lecture I will share with how I saved 80% of BigQuery monthly billing of investing.com.
How to reduce costs using GCP big Query? what should we pay attention to?
We are going to cover all of google best practices while working with BigQuery.

Video

Slides

27.10.2019

Video


——————————————————————————————————————————
I put a lot of thoughts into these blogs, so I could share the information in a clear and useful way.
If you have any comments, thoughts, questions, or you need someone to consult with,

feel free to contact me via LinkedIn – Omid Vahdaty:

NoSQL

High Performance, Low Latency Database Architecture

High Performance, Low Latency Database Architecture

Lecturer: Guy Shtub 14.3.2023

In this talk, I’ll speak about modern, distributed, high-performance databases. I’ll cover topics like architecture, consistency, high availability, replication, and scaling. As an example, I’ll use ScyllaDB however the concepts hold for Apache Cassandra as well as other Column Family databases based on the BigTable paper published in 2006.

Lecturer: Guy Shtub is Head of Training at ScyllaDB

Video

Slides


——————————————————————————————————————————
I put a lot of thoughts into these blogs, so I could share the information in a clear and useful way.
If you have any comments, thoughts, questions, or you need someone to consult with,

feel free to contact me via LinkedIn – Omid Vahdaty:

architecture

Data Mesh: Experimentation to Industrialisation

Data Mesh Architecture: Experimentation to Industrialisation

Lecturers: Sunny Jaisinghani and Simon Massey 27.2.2022

Discover what happened when a large financial service organisation who were already underway with a DevOps and Agile transformation went from a Monolithic Data Lake architecture, onto a federated self-service Data Mesh on Google Cloud Platform (GCP).

The key driver from the transformation was to reduce Lead times and improve the Flow Efficiency for Business Change. The typical approaches to transformation demonstrated substantial efficiencies across the core operational platforms but no material impact was seen on the downstream Data Publishing and Data Analytics platforms. These were faced with more fundamental blockers around lack of autonomy, monolithic architecture and proxy ownership of the data, compounded by legacy tech estate of on-prem data warehouses, data marts, data lakes, etc. End to end solutions required coordination between specialised teams working in silos leading to extended lead times.

This required a paradigm shift on both the systems architecture and Ways of Working.

In this session, we’ll explore the key driving principles for the Data Mesh from MVP, to productionisation to industrialisation.

The Data Mesh was built to be an Open Self Service platform whereby the various tenants can contribute to the features themselves alongside using the Core Platform self-service features. The success of the Data Mesh led to buy-in across the business and the Data Mesh Adoption accelerated exponentially. During the talk, we’ll highlight some of the key outcomes and business value delivered through the Data Mesh including:

  • Rapid business values delivered to many ongoing programmes building ML models, MI Dashboards, cross-domain analytics, Data Provider APIs, Enquire and Reporting apps, etc.
  • Teams were able to react to fast changing business and client demand with lead times dropping from months to days.
  • New business models identified.
  • The Data Mesh brought parity across the varying levels of technology maturity and skills within the organisation.

    The Data Mesh is now a de facto part of the downstream data publishing, reporting and analytics for the organisation.

     

    Who should watch?
    Anyone who wants to understand how Data Mesh can help businesses achieve their organisational objectives.

    What you’ll learn?

    • What is Data Mesh.
    • The key driving principles.
    • How the hyper-new concept delivers business value.
    • How Data Mesh works across different programmes.


    Lecture Langauge: English


    Speakers:

    Sunny Jaisinghani- Data Mesh Platform Owner
    Simon Massey- Data Mesh Lead Technologist

Video

Slides


——————————————————————————————————————————
I put a lot of thoughts into these blogs, so I could share the information in a clear and useful way.
If you have any comments, thoughts, questions, or you need someone to consult with,

feel free to contact me via LinkedIn – Omid Vahdaty:

Uncategorized

Keep your data encrypted in BigQuery

Keep your data encrypted in BigQuery

Lecturer: Ran Tibi 6.2.2022

If you work with data and build a data warehouse you probably have some sensitive data that you want to keep secured.
While BigQuery encrypts all the data before it is written to the disk, once you have read access to the tables you can have full visibility of the data, including sensitive data and PII.
In this talk, we will describe a use case that shows how you can create a secure end-to-end process that encrypts at the application level the data before inserting it into BigQuery and allow users to decrypt it only in query time without the need of knowing the actual encryption key.

Video

Slides


——————————————————————————————————————————
I put a lot of thoughts into these blogs, so I could share the information in a clear and useful way.
If you have any comments, thoughts, questions, or you need someone to consult with,

feel free to contact me via LinkedIn – Omid Vahdaty:

NoSQL

Next Generation Databases: Critical Innovations for Performance at Scale

Next Generation Databases: Critical Innovations for Performance at Scale

Lecturer: Guy Shtub, Head of Training at ScyllaDB, 15.12.2022

Modern database systems and how this plays out in current-day applications. Covering topics like consistency, availability, scaling, NoSQL vs SQL, and showing some examples with a look to the future.

Video

Slides


——————————————————————————————————————————
I put a lot of thoughts into these blogs, so I could share the information in a clear and useful way.
If you have any comments, thoughts, questions, or you need someone to consult with,

feel free to contact me via LinkedIn – Omid Vahdaty:

architecture

The lack of communication between data consumers

The lack of communication between data consumers

Lecturer: Noy Twerski, 15.11.2022

At this meetup, we’ll explore how poorly communicated companies can waste time and make mistakes in their business decisions.
We will also share best practices from companies that have already changed their collaboration culture and some tips on how to avoid the mistakes by collaboration methods.

Video

Slides


——————————————————————————————————————————
I put a lot of thoughts into these blogs, so I could share the information in a clear and useful way.
If you have any comments, thoughts, questions, or you need someone to consult with,

feel free to contact me via LinkedIn – Omid Vahdaty: