Augmented Analytics, the Remedy for Too Much Data

PinIt

By adopting an augmented approach to analytics, businesses will finally be able to make use of the rich, wide data they collect changing “too much data” from a problem to a real opportunity.

As the price of data warehousing continues to fall, companies can afford to track almost every business metric at an increasingly fine granularity. And, along with these metrics, companies can afford to capture more useful context about each individual transaction, customer, and product use. The key to acting on this mountain of data may be augmented analytics.

In theory, this data gives every organization the opportunity to develop a data-driven culture. The data should guide teams towards a better understanding of what actually drives loyal, satisfied customer behavior, what new products and services to offer, and how they should optimize every customer interaction.

See also: Augmented Analytics Benefits and Use Cases

However, this influx of data presents a new challenge: where to prioritize efforts? Without a clear direction for where to look first, what to investigate, or how to uncover insights, the opportunity is obscured by its complexity and dimensionality. Without a different approach, today’s enterprise data is challenging to operationalize.

Data width, not depth, is the new performance bottleneck for analytics. With more columns and features to analyze, this rising complexity creates three big blockers that prevent teams from getting the ROI they expect from their data: data prep and discovery, insight generation, and insight explanation.

However, a more augmented analytics stack can remove these barriers by shifting the burden of these tasks away from individual analysts. Not only does this accelerate time to insight, but it frees up data teams to do the more valuable work of synthesis, recommendation, and creative thinking.

Shifting the target: Analyzing wider, more useful datasets

Before augmented analytics, analysts could spend as much as 80% of their time transforming data into simplified datasets that BI tools and legacy data warehouses could computationally handle. This process put the burden on data engineering to downsample and aggregate the data to deliver more responsive reports and dashboards.

With the rise of cloud data warehouses that provide more scalable data storage and nearly infinite compute, dimensionality, and volume are no longer meaningful bottlenecks. Now, instead of spending time transforming the wealth of data businesses collect into a clean, aggregated format, data teams can shift their focus to creating richer, more comprehensive, more useful datasets that inform business decisions.

To properly diagnose a single metric, it’s typical for the data to come from three or more sources, such as a CRM system, a product analytics system, and a first-party database. Now that tools exist to augment data prep and bring together disparate data sources, analysts can intelligently and efficiently use all their data. This leaves the most crucial part of the analytics ecosystem – the analysts themselves – to do what they’re great at interpreting data insights to inform the business.

Accelerating insights: Get from the “what” to the “why” faster

An analyst’s unique skill is in defining metrics and testing hypotheses that surface actionable insights for the business. But it can be a manual, repetitive, and frustrating process that’s repeated for every new request.

As the sources and complexity of data grow, this manual approach becomes intractable, leaving analytics teams reactive to the business rather than proactively identifying opportunities for impact. Every new column and incremental factor adds thousands of possibilities to check. Combined with growing data volumes and increased urgency from the business, data teams are stuck in the middle.

With wider, deeper data, this time crunch means analysts often spend their time providing insights on what changed in the metrics and rarely have time to answer why in time for the business to act. The answer is not to hire more analysts to write queries and keep up with the ever-growing request for data; instead, it’s to help analysts get answers faster, with tooling that augments these analyst’s human creativity, intuition, and drive.

Augmented analytics tools allow analysts to surface useful, relevant insights as quickly as possible by making sense of the data and finding the factors or combinations of factors driving changes in their data. With faster answers, analysts can interpret the results and spend more time proactively finding the problems that need to be solved for the business — allowing analysts to explain not only “what” changed but “why.”

Streamlining explanation: Drive collaboration and improve data literacy

If the rest of the business can’t readily digest the insights data teams find, it doesn’t matter how quickly the answers arrive. Companies have tried to solve this problem through so-called self-service analytics solutions, but for most teams, these tools only serve to shift the burden of query generation to less data-savvy leaders, creating more work for more people but not addressing the core issue of making this data actionable. Even in organizations with very high data literacy, this is like asking the sales team to design and develop the product, too. 

Augmented analytics take a different approach to transforming the reporting and monitoring process, moving beyond spreadsheets and point-and-click static dashboards. Instead of relying on users to generate hypotheses and ask questions, augmented tools proactively stack-rank insights by relevance and impact. This approach not only provides critical insights faster but also explains why specific results are returned.

By presenting easy to understand facts—and focusing on the metrics that matter—augmented analytics allows different audiences across the business to collaborate on insights. This is the core shift needed for organizations to finally recognize a positive ROI from their data investments and build a lasting, competitive data-driven culture.

Put all your data to use and using an augmented analytics process

There are strong parallels between the way enterprise data is exploding today and the rise of the consumer internet. By 1997, the breadth of information available on the web had outstripped the ability of category-based indexes like Yahoo! to keep up. The internet needed a new interface, and search engines like Google stepped forward.

Enterprise data needs a new interface as well, one designed for the speed, breadth, and volume of the data collected by companies daily. Augmented analytics platforms are delivering that new interface, accelerating our ability to search, combine, analyze, and collaborate on insights quickly and comprehensively, without the need for complex SQL queries and manual data manipulation.

By adopting an augmented approach to analytics, businesses will finally be able to make use of the rich, wide data they collect — changing “too much data” from a problem to a real opportunity.

Peter Bailis

About Peter Bailis

Peter Bailis is the founder and CEO of Sisu, the fastest and most comprehensive augmented analytics platform for structured data. Peter is also an assistant professor of Computer Science at Stanford University, where he co-leads Stanford DAWN, a research project focused on making it dramatically easier to build machine learning-enabled applications. He received his Ph.D. from UC Berkeley in 2015, for which he was awarded the ACM SIGMOD Jim Gray Doctoral Dissertation Award, and holds an A.B. from Harvard College in 2011, both in Computer Science.

Leave a Reply