Backend Data Integration Challenges in the Age of Cloud

There is great optimism about the capabilities of digitally driven enterprises. However, even the most digitally savvy enterprises may find this optimism tempered by the challenges of data integration that is essential to decision-making.

Even with the rise of cloud capabilities and toolsets, enterprises are still struggling to find their equilibrium with data integration. Outdated tools and technologies make it difficult to get timely information to business users, according to a study from TDWI. “Organizations struggle to make informed decisions, achieve smarter customer engagement, and enjoy agile and efficient operations without good data integration,” according to David Stodder, research director with TDWI and author of the study. “As critical as data integration technology is to how fast businesses can react to customers and market needs, the survey indicates many organizations are not there yet.”

Traditional ETL continues to reign as the most oft-employed method of data integration, cited by 73%. Another 35% have moved to real-time replication and/or change data capture. Just under a fifth (18%) were making use of data virtualization or federation. There are still too many spreadsheets being employed to attempt to cobble together information. As a result, 57% say their data integration efforts are too slow to keep up with business priorities.  

Much of the emphasis around data integration has been seen in the ongoing movement to cloud-based infrastructures, the survey adds. The majority of respondents (79%) plan to expand data management for business intelligence and analytics currently on cloud platforms. Only one percent plan to decrease such use. “Many organizations are shifting the focus of data integration to deliver data quickly to cloud platforms from both new data sources and legacy systems,” Stodder notes. Cloud is increasingly supporting faster and more comprehensive methods of integration, including automated data pipelines, real-time data streaming, and data lakehouses that integrate cloud data warehouses and data lakes.

See also: Is the Data Cloud Alliance for Data Openness or for Google?

The survey’s results point to some overarching trends accompanying the migration to the cloud:

There is a strong desire for a centralized data integration architecture. Having a single data architecture is a goal endorsed by 72%, who agree that centralized integration platforms provide opportunities to “provide more options for managing an increasingly diverse range of data structures, end-user types, and business use cases.” Only 11% felt that a centralized environment would create more problems “because of the complexity of the resulting architecture and the work required to build a data lake and/or modernize a data warehouse.”

Cloud-based data integration means more than analytics. “For some organizations, migration to the cloud as part of the digital transformation of business applications and operations is at least as important as analytics,” Stodder points out. “Digital transformation typically increases and enhances the role of data; both users and automated functions need continuous and timely data to drive smarter decisions. Data integration must be agile to address daily operational data needs as well as those for data science and analytics.”

Some organizations still prefer to keep analytics on premises. Security and availability are still top of mind when it comes to key data assets. As a result, many executives “worry about hacking, unauthorized access, and potential service outages with cloud provider platforms, even as these platforms display increasingly dependable security and availability,” says Stodder. “If these concerned organizations use the cloud, they often prefer private cloud arrangements.”

There is a major commitment to the cloud among organizations that say analytics is their most significant driver. Four out of five of these respondents (80%) have cloud data warehouses, and 65% have cloud data lakes. Nearly three out of five (59%) expect their cloud-based data management to increase significantly in the next 12 months.