How Change Data Capture Can Speed Development and Deployment

In any application modernization project, businesses that choose to add CDC to their plans from the beginning will reap the rewards, again and again, as they react to the competition.

Application modernization is the new must-have for large businesses with many legacy applications that need to keep up with disruptive, emerging competitors. CTOs, CIOs, and key engineering stakeholders are all-in on overhauling the entire software development lifecycle—with a particular focus on development and deployment—to push new features faster than ever. Change Data Capture has an important role to play.

One of the ways these businesses are tackling their modernization efforts is by migrating from monolithic deployments to event-driven architecture (EDA). Unlike legacy applications, which rely heavily on “pulling” data from a database whenever it’s updated, EDA uses producers to publish events into an event stream. Other parts of the application “listen” to the event stream by subscribing to it and immediately take action on any relevant events.

An EDA comes with a ton of built-in benefits—they’re built with today’s technology, which means it’s easier to hire the right talent to continue developing and deploying. They’re cloud-native, containerized, scalable, streaming-first, and support more sophisticated analytics. But if businesses go all-in on EDA without preparing for the potential hiccups, they won’t be able to unlock the speed they’re after.

Can Change Data Capture reduce the modernization risk?

Modernizing a complex legacy application doesn’t guarantee faster development or deployment speeds. In our interview with Bilgin Ibryam, product manager and a former architect at Red Hat, he outlined four phases of modernization:

• Rehosting: Take an existing application and move it to a new machine/provider.   
• Re-platforming: Make development changes to the “edge” of an application, such as its front-end UI, to take advantage of the features of a particular cloud platform.
• Platform changes: Move to a container-based application strategy.
• Re-factoring: Split a legacy application into modular, separate microservices based on the business domain and function.

When modernizing an application, the business is actually doing all four of these phases. They’re moving to a new deployment structure, choosing a cloud provider, and developing features based on new APIs.

As Ibryam puts it: “[The business] may not realize that they are creating the blueprint, the new best practices, and the new architecture of their future microservices.”

It’s critical to find the right pieces at the beginning of the process to avoid complex and difficult-to-revert changes that affect the team’s ability to develop and deploy new features at the competitors’ speed. And a lot of teams looking into EDA for their modernization—and are keen on reaping the benefits in speed—are also opting in on Change Data Capture (CDC).

Related: Why Change Data Capture is Essential to Implement Event-driven Architectures

CDC is a system that monitors and captures just changes in data so that other parts of the EDA can respond accordingly. When one part of the application inserts, updates, or deletes database rows or tables, the CDC adds to the existing event stream, ensuring that downstream consumers know that there’s work to be done.

A basic version of a CDC implementation uses timestamps to convey the last time another part of the application changes a given record. For example, when a user first creates an account, their record populates the last_updated column with the current time. If they change their email address associated with their account the following day, either the application logic or a database trigger updates the column again.

Now, instead of pushing an entire record into the event stream for further processing by a different consumer, the CDC makes the change in data—the fact that the record was changed at a specific time—available to the event stream. This simplifies the entire architecture and minimizes the amount of data “on the wire” between various microservices.

What happens when EDA and CDC meet?

One popular combination of EDA and CDC uses Debezium and Kafka to migrate data and streaming events between microservices. Debezium is an open-source CDC platform that monitors database logs to identify changes, which means that it doesn’t even need write access to the database itself—simply allow Debezium to access the logs, and it can get to work capturing change and adding to the event stream.

This dramatically simplifies the process of developing and deploying the modernized application for the first time. There’s no worry over destroying the legacy database—just filter out the required columns with the CDC and put those into the event stream.

Subsequent development gets faster, too, because teams can build new features on microservices that react to only the essential change-based events.

For example, using the updated timestamp example from before—let’s say there’s an application that shows the user updates to their account since the last time they logged in. Instead of requesting records from an arbitrary timeframe, the last 24 hours, for example, it can poll the database based on the update timestamp, which means it always picks up the smallest payload of new information.

By developing and deploying around change, not entire database rows, developers can focus their efforts on the most relevant data, with the least impact on the event stream or those all-important databases. If any application modernization project is a blueprint, businesses that choose to add CDC to their plans from the beginning will reap the rewards, again and again, as they react to the competition.