Enabling Financial Services Real-time Time-Series Analytics

Fraud detection, wealth management, regulatory compliance, and portfolio optimization are among the many financial services disciplines that improve when services in these areas are delivered in real time. To accomplish this requires real-time time-series analysis.

Real-time time-series analytics allows financial services institutions to monitor and assess the impact of billions of daily market transactions. And it allows them to constantly update investment positions while considering risk and exposure.

One of the major challenges that has limited the use of real-time time-series analytics is that existing analytics infrastructures were not designed for real time. In a traditional approach, data is batched, then processed as transactions. The transactional data is then remixed via ETL and placed in an analytics database or data warehouse. The analysis is done then.

That introduces enormous time delays from an event happening (e.g., a transaction is made or a financial indicator changes) to the time the organization has insights upon which they can take action. In today’s real-time world, that delay is not acceptable. Fraud detection, risk assessment, portfolio recommendations, and more must be done in real time.

See also: Booming Data Volumes and Velocities Require Vectorized Databases for Real-Time Analytics

An infrastructure for real-time time-series analytics

To keep up with the speed, scale, and complexity of financial data and deliver differentiated services to customers requires active analytics that unifies streaming, historical, and location analytics with machine learning on a single platform. Such a solution can power real-time trade and risk decisioning applications financial services institutions need to increase revenue and improve compliance.

Real-time fraud detection and risk assessment would benefit from such an infrastructure. Why? Most data systems, even modern ones, are poorly designed to handle the high-cardinality joins and the constant aggregation and re-aggregation to continuously re-evaluate positions.

Stream processing tools lack the context of holdings and reference data. Batch processing with traditional data systems is too slow. Trying to combine the two by stitching together specialized tools using lambda/kappa architectures is just too complex. And these solutions lack specialized capabilities such as temporal joins and ASOF joins.

A better solution is to use a real-time analytics database that fuses streaming data with static reference data for a full and constantly updated picture of positions with context on risk and exposure. Additionally, there are critical elements needed to enable real-time time series analysis. For example, a suitable solution might use a lockless architecture and vectorized compute algorithms. That would allow for continuous re-aggregation as streaming data changes and enable an organization to do complex analysis on demand using the most up-to-the-moment data.