High latency in applications built on data warehouses creates a poor user experience and presents a significant challenge for developers.
Data warehouses use OLAP for complex, low-volume queries on large datasets, contrasting with OLTP's high-volume, simple transactions.
User-perceived performance is impacted by network delays and data scan times, making sub-second responses a critical goal.
BigQuery's cache only works for identical queries and its concurrency is capped per project, impacting real-world application performance.
Load testing reveals that BigQuery maintains a consistent query latency of around two seconds regardless of user concurrency up to its hard limit.
Cube provides a semantic layer over data warehouses, enabling caching, pre-aggregations, and access control to build fast data apps.
A local Cube instance can be configured using Docker Compose to connect to BigQuery and automatically generate data schemas.
Pre-aggregations act as materialized views that store condensed query results, reducing a query's response time from seconds to milliseconds.
Benchmarks show that using Cube's pre-aggregation layer results in a nearly five-fold performance increase over querying BigQuery directly.
The discussion covers when to implement a caching layer, how Cube improves performance, and its utility for medium-sized databases.
24:22Jürgen Pilz
27:27Michele Riva
28:12Josip Stuhli
50:06Jens Happe
42:08Matteo Collina
22:32George Asafev
29:25Matthias Niehoff
30:02Abdelrahman Awad
.gif?w=240&auto=compress,format)
Jobs that call for the skills explored in this talk.
CodeGuild
Utrecht, Netherlands
Scalefree Scalefree