Splunk Aims to Drive Real-time Analytics

Splunk is now starting to move well beyond batch-oriented analysis of logs to apply analytics to streaming events as they occur in real time.

To accelerate that transition Splunk is moving to acquire Streamlio, a provider of an open source distributed messaging platform that includes tools for analyzing data in real time. That platform will soon be employed to extend Splunk Data Stream Processor (DSP) and Splunk Data Fabric Search (DFS) offerings that Splunk announced are both generally available as its recent .conf19 conference.  Splunk DSP is a real-time stream processing platform continuously collects high-velocity, high-volume data from diverse sources to facilitate analytics that Splunk CTO Tim Tully describes as “the secret sauce” on which the Splunk real-time analytics strategy for routing data is based.  

See also: Splunk Sees AI Moving from Early Adopters to Mainstream in 2019

Increasingly, that analysis is going to be performed using machine learning algorithms that will augment the capabilities of humans because not only is the rate at which data is being processed is increasing, so too are the types of data that need to be compared and contrasted, adds Tully. Rather than just monitoring logs, Splunk will be able to track and surface a wide range of metrics in real time, says Tully.

The decision to build Splunk DSP and acquire Streamlio is being driven by the increased need to respond to events in real-time, says. For example, Tully says Splunk sees embedding Streamlio both within its security operations and automation response (SOAR) platforms and the application performance management (APM) platform it gained by acquiring SignalFx to analyze events in real-time.

At the same time, however, Splunk is also moving beyond IT operations analytics to surface business insights to end users. As organizations launch various digital business transformation initiatives it’s already apparent much of that data will need to be analyzed in real time, says Tully.

“It’s all part of the consumerization of the enterprise,” says Tully.

In fact, Splunk already makes available Connected Experiences platform that leverages natural language processing to verbally interrogate data using Splunk Mobile and Splunk TV apps. Splunk also makes available a Splunk Business Flow tool to discover and mine the workflows that make up a business process.

The challenge IT organizations are increasingly encountering is there is no time to extract, transform and load (ETL) data into some normalized format to run analytics. Rather, actionable insights that can drive an automated process need to be surfaced using real time using platforms designed specifically of those use cases, says Tully. Those use cases don’t obviate the need for the existing Splunk platform, but organizations are looking for a single vendor to address a range of data analytics requirements, notes Tully.

It’s too early to say to what degree Splunk will be able to deliver on its ambitions to deliver data to everything. What is clear is that most of that strategy hinges in one form or another on the ability to surface analytics across a broad range of use cases in real-time. After all, all the analytics in the world isn’t worth much if it can’t be used to alter a business outcome.