Want AI? It’s Going to Require a Healthy Dose of Real-Time Data Streaming

Everyone wants to compete with artificial intelligence these days, and there’s nothing wrong with that. There’s a catch, however. For AI to function properly, organizations will need to shift away from the batch processing loads of data they handle, toward a real-time streaming model. And that hasn’t been easy.

That’s the word coming out of a survey of 300 engineering organizations currently working with streaming data, published by Redpanda. The survey finds real-time analytics and AI are driving streaming data adoption. At least 75% of survey participants were at various stages of adoption.

Real-time analytics (71%) is the leading current use case for adopting streaming data systems. Looking forward, nearly three in four respondents cited development of AI/ML systems will be the biggest driver of streaming data adoption in the next 12 to 24 months.

There are some well-dug-in obstacles to overcome, however. “Organizations today increasingly look to enrich their applications, analytics platforms, and AI/ML models with real-time data, which requires a transformation from traditional batch processing to streaming data systems that process and analyze gigabytes of data per second,” the report’s authors state.

See also: Introducing the Data-in-Motion Ecosystem Map

“In streaming data pipelines, real-time data flows continuously as it’s generated from sources such as sensors, devices and applications. However, streaming data adoption varies across industries and streaming data systems can be challenging to implement, manage and scale for the unprepared.”

Typically, streaming organizations are hybrid, the survey shows. Platforms in use encompass both Apache Kafka-compatible and Kafka non-compatible solutions. More than half of current users stated that their data streaming infrastructure is hosted on virtual machines or containers and is located in a hybrid environment. AWS (57%) and Microsoft Azure (57%) were the most common cloud providers selected.

“AI and machine learning are going to shake up how companies operate, and streaming data amplifies the leverage that can be gained from these powerful tools in real-time for bigger impact,” the report’s authors state.