Booming Data Volumes and Velocities Require Vectorized Databases for Real-Time Analytics

Companies are awash in data that, if analyzed soon after it is generated, can vastly improve operations and deliver significant value to the organization. The challenge most face in making this happen is that traditional methods of analyzing data were not designed to handle the volume and speed of the data. Vectorized databases may be what’s needed.

To put the issues into perspective, consider what’s changed with the data itself. To start, IoT data, web data, business transaction data, and more are being generated at massive rates. IDC estimates data generated from connected IoT devices alone will be 73.1 ZB by 2025, growing from 18.3 ZB just a few years ago.

It is not only an issue of there being more data sources. The bigger issue is that each source produces more data than ever before. For example, in the past, an industrial IoT sensor might make one measurement, such as the speed of a motor or the system’s temperature. And it might make that measurement once a minute or once an hour. Today, a smart sensor is much more likely to measure multiple metrics about a device. And those measurements are being made on a much shorter time scale of a second or even faster.

Additionally, not only does IoT and sensor data require real-time processing to extract value, it also requires temporal and spatial joins to provide context.  Conventional database architectures were never designed to address these two things (real-time processing and temporal / spatial joins).

Combined, these factors require new thinking on how to analyze the massive streams of data being generated all the time. Companies need a way to quickly recognize patterns in the data, combine that real-time data with historic data and derive insights that allow decisions to be made and actions taken soon after the data is generated.

The need for speed

Time relevancy is more important than ever. Most high-value IoT use cases require decisions to be made in real time. The same is true with other data. A retailer needs to react to a customer’s journey through an online store and offer personalized suggestions while that person is moving around the site. A security operations team must make split-second decisions about nefarious actions to prevent cybercrime from happening.

The bottom line is that real-time decisions informed by real-time data are increasingly critical in many industries and across many application areas. Quite often, the ability to make such decisions relies on the quick analysis of multiple data sources with vastly different attributes. The essential datasets upon which the analysis must be performed typically include growing volumes of streaming data that contain key events combined with historical data that provides context.   

Commonly used methods simply break down or do not deliver the real-time insights needed today. For example, data warehouses and data lakes incur both data latency and query latency. Both add to delays that prevent a company from taking immediate actions based on real-time analytics.

Streaming technologies like Apache Kafka cannot incorporate historical and contextual data. When used alone, these technologies operate as a data queue, where data is moved from one system to another in real time. Since the data is not persisted, there is no accumulation of history or ability to join with other data. Further, they often have limited time-series analytics capabilities.

See also: Real-Time and Automation: Trends to Watch in 2023

Enter vectorized databases

Current generation databases use conventional CPUs that sequentially perform calculations. Most can support parallel operations by distributing work across multiple nodes. However, there is a bottleneck occurs at the processor level.

Breakthroughs in database architectures have given rise to a new generation of databases that vectorize the calculations within the processor, acting as a force multiplier on performance. They offer order-of-magnitude performance improvements to doing temporal joins across big data sets that simply were not possible with traditional massively parallel processing (MPP) databases.

More recently, organizations have begun to speed analysis by employing GPUs or CPUs that use Intel’s Advanced Vector Extensions (AVX) so that operations can be applied to all the data in a database in parallel. As such, databases based on vector processing are gaining wide popularity in that they efficiently use modern chip architectures on commodity hardware and in the cloud to optimize analytic performance.

Vector instructions can include mathematics, comparisons, data conversions, and bit functions. As such, vector processing exploits the relational database model of rows and columns. That also means columnar tables fit well into vector processing.

What vectorized databases do best is they step through a large block of memory with a list of instructions. A GPU or Vectorized CPU can easily compute massive amounts of time-series data directly. Additionally, aggregations, predicate joins, and windowing functions operate far more efficiently with vectorized databases.

A final word

Vectorization unleashes significant performance improvements and cloud compute cost savings – particularly on queries at scale. Aggregations, predicate joins, windowing functions, graph solvers, and other functions all operate far more efficiently.

The result is that companies using vectorized databases can derive actionable insights from their data streams in a time frame to take immediate actions as events occur.