Change Data Capture and the Many Roles it Can Play

Change Data Capture (CDC) is a data integration pattern to track when and what changes occur in data then alert other systems and services that must respond to those changes. CDC helps maintain consistency and functionality across all systems that rely on data.

In modern microservices-driven architectures, CDC has gained new importance by providing a bridge to connect traditional databases with cloud-native, event-driven architectures. Using CDC, enterprises can continue to use their legacy databases while still making use of data through emerging technologies. For new deployments, CDC enables the use of useful patterns and schema like the “outbox,” which allows microservices to exchange the consolidated data from a database transaction.

With such capabilities, CDC can play many roles in application development and modernization. Here are several use cases for CDC. 

See: Why Change Data Capture is Essential to Implement Event-driven Architectures

Serve as the cornerstone of legacy app transformation

Many monolithic applications in many industries like banking, retail, hospitality, logistic, and others are event-driven. Traditional applications with tight coupling between components rely on request-driven messaging processes, where calls for data are made directly from one service to another. In monolithic application environments, these processes are tightly integrated and run on the same system.

As monoliths are broken up into microservices, the event producer and consumer may be in different microservices. The ability for one microservice to publish events to others is a necessity to carry forward the core programming constructs into a distributed computing architecture.

That’s where Change Data Capture comes in. CDC is a system that monitors and captures the changes in data so that other software can respond to those changes. CDC captures row-level changes to database tables and passes corresponding change events to a data streaming bus. Applications can read these change event streams and access the change events.

Speed application development and deployment using Change Data Capture

One of the ways these businesses are tackling their modernization efforts is by migrating from monolithic deployments to event-driven architecture (EDA). And they are using CDC to monitor and capture just changes in data so that other parts of the EDA can respond accordingly.

One popular combination of EDA and CDC uses Debezium and Kafka to migrate data and streaming events between microservices. Debezium is an open-source CDC platform that monitors database logs to identify changes, which means that it doesn’t even need write access to the database itself—simply allow Debezium to access the logs, and it can get to work capturing change and adding to the event stream.

This dramatically simplifies the process of developing and deploying the modernized application for the first time. There’s no worry over destroying the legacy database—just filter out the required columns with the CDC and put those into the event stream.

Subsequent development gets faster, too, because teams can build new features on microservices that react to only the essential change-based events.

Scale streaming projects with Change Data Capture

Change Data Capture helps organizations modernize applications and efficiently scale them. Using a microservices approach to modernizing and building applications, organizations can make fast changes to existing front-end applications and create new applications. With change data capture software consuming fewer resources as it only addresses data changes, these applications can retain their performance and scale without a massive infrastructure build-up.

CDCs fit well with one additional industry trend. The composable enterprise, where code that enables a single process or step is reused in different applications, is on the rise. Building on the flexibility of cloud-native development, here, specific processes or business functions are standardized and can be reused in numerous applications.

Enable real-time applications

Change Data Capture lets any type of application take advantage of having low latency when it comes to reacting to data changes. This is an essential element in real-time systems. One specific example of its use is to keep a search index up to date so people can do a full-text search taking into account any real-time updates. It can also be used as an on-ramp to the cloud for on-premises legacy databases. For example, it could be used to take data from an on-premises legacy database and insert it into a cloud data warehouse. In this use case, CDC lets a business set up such a data pipeline, and it would have a very low end-to-end latency.

When updating a source database, which could be any relational database such as Oracle, Microsoft SQL Server, Postgres, or MySQL, a business may need to update multiple related resources such as a cache and a search index. A simple approach would require upgrading the applications to update those resources at the same time. However, trying to consistently write this changed data to more than one target introduces many challenges and coordination overhead. CDC avoids issues like dual writes to, instead, update resources concurrently and accurately.