ING's initial centralized data lake, which used a common format, struggled with manual ETL processes and scalability issues across a multi-cloud landscape.
The evolution of the flower industry from a centralized to a decentralized logistics model illustrates how data can move directly while metadata remains coordinated.
The data mesh offers a powerful decentralized architecture, but its implementation is a significant engineering challenge as no single product exists.
ING collaborates on open source projects like Fabric for the data mesh and identifies the need for a separate metadata mesh to solve interoperability issues.
Egeria is an open source project under the Linux Foundation that provides a standard for exchanging metadata between different vendor tools via a two-layer API.
ING's dedicated team contributes directly to the Egeria open source project, navigating the challenge of prioritizing a community-driven backlog over internal requests.
ING leverages Egeria for unified metadata search and to automate end-to-end data lineage, which is crucial for meeting regulatory requirements like BCBS 239.
The Q&A session covers incentivizing data ownership through monetization, motivating metadata provision, and comparing Egeria's comprehensive model to other industry solutions.
37:21Mario Meir-Huber
29:25Matthias Niehoff
32:12Clemens Vasters
29:31Adam Larter
21:17Lars Kölker
20:04Kateřina Ščavnická
28:29Taqi Jaffri, Kapil Gupta & Farooq Sheikh and Tomislav Tipurić
19:44Dominik Schneider
.gif?w=240&auto=compress,format)
.gif?w=240&auto=compress,format)
Jobs that call for the skills explored in this talk.
