Kafka is a distributed event streaming platform using topics, partitions, producers, and consumers for scalable data processing.
The architecture evolved from traditional request-response microservices to an event-driven model using Kafka as the single source of truth to improve decoupling and extensibility.
The system uses an API service with materialized views for robust reads and command processing topologies that can recover from failures by replaying input topics.
A data pipeline cleans, deduplicates, and amends product data from various sources, then streams it to Elasticsearch to create a searchable materialized view.
A command processing pipeline implements the CQRS pattern by separating write operations from read models, using an event topic as the source of truth for inventory data.
Race conditions caused by eventual consistency are solved by using manually updated state stores and repartitioning command streams to ensure data locality for validation.
New features like automatic warranty extensions can be added by deploying new services that consume existing data streams without modifying the original producers.
A pure Kafka Streams architecture presents challenges in development complexity, stateful operations, careful configuration for transactions, and operational tooling.
The architecture can be evolved by integrating traditional databases to simplify complex stateful logic, while using connectors to publish all state changes back to Kafka.
46:24David Leitner
30:51Soroosh Khodami
38:50Lucia Cerchie
54:29DeveloperSteve
24:22Jürgen Pilz
45:36Clemens Vasters
20:28Bastian Heilemann & Akash Manjunath
44:37Natan Silnitsky
.gif?w=240&auto=compress,format)
Jobs that call for the skills explored in this talk.
evoila Frankfurt GmbH
Mainz, Germany
SYSKRON GmbH
Regensburg, Germany
smartclip Europe GmbH
Hamburg, Germany
AUTO1 Group SE
Berlin, Germany
Infosys Limited
CONTIAMO GMBH
Berlin, Germany
