Bringing Clarity to Event Streams: Enabling Analytics and AI Through Rich Metadata
What if you could define a schema for data you don't have yet? Learn how to formalize event streams for powerful, AI-assisted tooling and analytics.
#1about 3 minutes
The problem of ambiguous data without clear context
Examples with LLMs demonstrate how data without metadata, like temperature units, leads to inconsistent and unreliable interpretations.
#2about 5 minutes
Using schemas to provide necessary data constraints
Providing a schema with explicit constraints and descriptions, such as units, enables LLMs to interpret data correctly and generate consistent results.
#3about 1 minute
Adopting a schema-first design for data structures
Data schemas should be created first and be independent of code assets, allowing code generation for polyglot applications and improving LLM performance.
#4about 3 minutes
Standardizing event structure using CNCF CloudEvents
Using a standard like CloudEvents provides a consistent envelope for events, preventing the chaotic and varied structures that LLMs might otherwise generate.
#5about 5 minutes
Automating client code generation with detailed metadata
By providing a comprehensive metadata document that includes endpoint and protocol details, LLMs can automatically generate complete and correct client code for publishers.
#6about 2 minutes
Moving beyond tribal knowledge in event pipelines
Current event pipelines often rely on informal communication, but a metadata-driven approach creates a formal contract that improves maintainability and compatibility.
#7about 6 minutes
A practical metadata-driven pipeline with Microsoft Fabric
A real-world example processing USGS water data demonstrates how a complete metadata definition using X Registry enables automated tooling and contract enforcement.
#8about 2 minutes
Introducing JSON Structure as a better schema language
JSON Structure is proposed as a stricter alternative to JSON Schema, designed as a data definition language with a better type system and namespace support.
#9about 4 minutes
Key standards for modern event-driven architecture
A summary of three key open standards—CloudEvents, X Registry, and JSON Structure—that provide the tools for building robust, metadata-driven systems.
Related jobs
Jobs that call for the skills explored in this talk.
Matching moments
01:06 MIN
Creating a standard for micro-frontend metadata
Micro-Frontends Discovery
04:18 MIN
Why modern applications adopt event streaming
Event Messaging and Streaming with Apache Pulsar
03:06 MIN
Building an open source data mesh and metadata mesh
A Data Mesh needs Open Metadata
03:41 MIN
Decoupling microservices with event streams
From event streaming to event sourcing 101
04:18 MIN
Using streaming data to power real-time agent applications
Unlocking Value from Data: The Key to Smarter Business Decisions-
05:40 MIN
Evolving from classic microservices to event-driven design
Kafka Streams Microservices
04:49 MIN
Introducing Egeria as an open metadata standard
A Data Mesh needs Open Metadata
06:00 MIN
Q&A on data ownership, monetization, and competing tools
Dev Digest 134 - Where pixels sing?News and ArticlesWeAreDevelopers LIVE Data and Security Day is on Wednesday, 25/09/2024. Learn about OPC UA Updates, Best Practices for Using GitHub Secrets, Passwordless Web 1.5, Emerging AI Security Risks, Data Privacy in LLMs and get a chance to t...
Panel Discussion: Responsible AI in Practice - Real-World Examples and ChallengesIntroductionIn the ever-evolving landscape of artificial intelligence, the concept of "responsible AI" has emerged as a cornerstone for ethical and practical AI implementation. During the WWC24 Panel discussion, three eminent experts—Mina, Bjorn Brin...
Daniel Cranney
Stephan Gillich - Bringing AI EverywhereIn the ever-evolving world of technology, AI continues to be the frontier for innovation and transformation. Stephan Gillich, from the AI Center of Excellence at Intel, dove into the subject in a recent session titled "Bringing AI Everywhere," sheddi...
From learning to earning
Jobs that call for the skills explored in this talk.