An overview of how change data capture (CDC) with Debezium and Quarkus can solve the problem of dual writes in microservices.
Dual writes to multiple databases or services can lead to data inconsistencies when one of the writes fails.
Apache Kafka is a fault-tolerant, scalable, publish-subscribe system designed for real-time event stream processing.
Change data capture automatically captures database changes like inserts, updates, and deletes and streams them as events.
Debezium is a Kafka connector that taps into database transaction logs to reliably capture and propagate data changes.
Debezium change events are JSON messages containing before and after states of the data, plus metadata about the operation.
The outbox pattern ensures data consistency by writing business data and an event to an outbox table within a single database transaction.
The strangler fig pattern uses CDC to replicate data from a monolith to a new microservice, enabling a gradual and safe migration.
Use Quarkus to implement the outbox pattern and deploy the entire system, including Kafka managed by Strimzi, on Kubernetes.
A practical demonstration shows how inserting data into a database table automatically triggers Debezium to publish a change event to a Kafka topic.
Discussion covers the challenges of building a custom CDC solution, Debezium's fault tolerance, and handling lost transaction logs.
57:19Mary Grygleski
44:40Alex Soto
38:50Lucia Cerchie
39:10Niklas Heidloff
33:40Niklas Heidloff
39:16Holly Cummins
37:55Adam Bien
35:27Kirill Kulikov
Jobs that call for the skills explored in this talk.
AUTO1 Group SE
Berlin, Germany
evoila Frankfurt GmbH
Mainz, Germany
Infosys Limited
Akros Ag
Akros Ag
Cosonic GmbH
ADMIRAL Technologies