“Using AI to optimize BigQuery”
As we kick off 2019, one thing is certain: the “Serverless Revolution” has hit the mainstream. However, whereas a majority of serverless discussions tend to focus on its utility in software development, there is a parallel paradigm shift occurring in big data analytics.
With fully-managed data warehouses like Google BigQuery and Amazon Athena, users can start querying petabytes of data at unprecedented speed, paying only for the compute and storage used. Small startups to companies as large as Spotify and Home Depot are migrating to this new way of doing analytics every day.
However, it’s on-demand pricing can be a double-edged sword with data analysts and teams. Scan more data than is necessary, and your company will wind up with an expectedly large bill at the end of the month. Worst of all, with no real-time price transparency while querying, team’s don’t learn about this until it’s too late. As a result, companies set usage limits on their analysts.
This is not an efficient approach for data analytics services as powerful as BigQuery or Athena, as it sets a ceiling to an analyst’s potential contribution. The solution? SQL optimization.
Avi Zloof and Eben du Toit, CEO and Chief Data Scientist at superQuery, respectively, will discuss building ML models to optimize SQL queries and how that removes the need for any restrictions on data analysts.
https://www.youtube.com/watch?v=cXq2tffYQ-A
about the lecturers:
—————————-
Avi Zloof:
Avi has spent the past 20 years leading R&D and innovation teams, managing big data and full stack development initiatives.
Prior to founding superQuery he worked for six years at TradAir leading cloud and big data projects. Big data tools he’s developed are being used by Tier-1 banks daily to trade and analyze billions in currencies.
Ido Vollf:
Ido is a serial entrepreneur who previously founded Sleeve (Edtech), EmbraceMe (IoT), and ChaChange (FinTech).
Ido worked as a Technical and Business Evangelist at Microsoft and was in charge of communicating the value of Microsoft Azure and Windows 10 to Israel’s startup ecosystem.
He’s an active mentor in the Israeli startup ecosystem, helping early stage startups grow fast.
Eben du Toit: Storyteller, data scientist, tech geek and computer engineer.
I’m a storyteller, data scientist, tech geek, computer engineer and control systems expert. My data skills were honed conceptualising, implementing and leading the efforts in building data science stacks. During my engineering career I’ve designed, coded, installed and tested full-stack large-scale IT engineering infrastructure projects at several power plants across South Africa and developed software for both mobile back-end platforms and power industry applications. I have over 14 years experience in engineering and grew up as a child of the internet. Currently chief data scientist at superQuery.
contact me for more details or questions (would love to help)
- Omid Vahdaty
- +972–54–2384178
Want more big data quality content? Join our meetup, subscribe to youtube channels
- https://www.meetup.com/AWS-Big-Data-Demystified/
- https://www.meetup.com/Big-Data-Demystified/
- Big Data Demystified YouTube
- AWS Big Data Demystified YouTube
- https://big-data-demystified.ninja/
For more information about Superquery:
https://web.superquery.io/?camp=BDD
——————————————————————————————————————————
I put a lot of thoughts into these blogs, so I could share the information in a clear and useful way. If you have any comments, thoughts, questions, or you need someone to consult with, feel free to contact me: