Presentation: Practical Change Data Streaming Use Cases With Apache Kafka & Debezium

I work as a software engineer at Red Hat and there I'm the lead of the Debezium project, which is a tool for change data capture.

I would like to first and foremost familiarize people with the concepts and ideas of change data capture. What is this about? But most importantly, what are the use cases for it? Why would you like to use change data capture? There are many use cases like data replication, data exchange between microservices, you could use this to enable streaming queries or auditing. I would like to familiarize people with those concepts. It's liberation for your data. You have data sitting there in a database and CDC allows to react to changes in the data. This liberation of data, that's what I would like to talk about.

The idea there is that very often people have this requirement that they need to update multiple things from within their application. Let's say they need to process a purchase order. But at the same time, they would also like to update the search index or they would like to send a message to Kafka to notify any downstream consumers about the order. Typically those two things, the database and Kafka, they cannot be updated atomically within one global transaction. If you've tried to do this, you're bound to fail and you will end up with inconsistencies. This outbox pattern essentially is a way to avoid this. How it works? You don't only update  the business tables in your database, but within the same transaction you also insert an event record into the outbox table. You then capture the insert from the outbox table and stream these change events to downstream consumers.

In the end, you could say it's that. It essentially allows you to have this instant read your own writes for your own changes. You could go to the database and you would see this newly persisted purchase order. But then at the same time, it also gives you eventually consistent eventing to downstream consumers.

First of all, there is this notion of decoupling. This all will be asynchronous. Even if you cannot reach Kafka for some time, eventually you’ll  be able to send events to Kafka again, with the source application not being impacted by any downtime. Also, if any consumers of those events are not available, let's say our search index, we cannot access it for some reason, we are not bothered by that because Kafka's it's in there and decouples them. Then one of the things which I really like about Kafka is that it's like a durable log. We can keep change events in Kafka topics for as long as we want, and you could reread topics from the beginning. This means we could add a new consumer down the road long after those change events were produced. For instance, we could add a consumer which takes the data and writes it to a data warehouse. And maybe we didn't even think about this use case when we were producing those events originally.

Three things, mostly. One touches a bit on the outbox pattern. Friends don't let friends do dual writes. That's one of the points I would like to talk. The next one, people should get an understanding what it is in there for them if they would use change data capture. What are the use cases? How could they benefit in their jobs by using this? And then finally, I would like to run them through some practical matters. How could you use this on things like Kubernetes or what are typical topologies? Sometimes people would like to stream changes for secondary database in the cluster. Those practical matters.