How APM and AIOps Help Enterprises Cross the Big Data Skills Gap

Big data is maturing and has hit that point of inflection where an increasing number of Fortune 1000 enterprises are deploying business-critical applications that depend on the big data ecosystem. A few years ago, data lakes seemed like a curiosity, but now they’re a necessity. The issue now is how to fully operationalize and extract value from those data lakes. While enterprises are now clearly committed to big data, they’re struggling to maximize their investments as they find themselves without the necessary skills to properly troubleshoot and manage these deployments.

Anywhere you want to look, big data skills are in short supply. Some of the biggest corporations struggle to hire and retain enough experts to tame and optimize their big data. Technologies like Spark, Hive and Kafka, are very complex under the hood – and when something goes wrong, users must comb through massive amounts of data to determine the cause and find a fix. It’s difficult for big data specialists to do this, and impossible for the average IT or business analyst.

See also: For DevOps, how to engineer a real-time feedback loop

Luckily, there’s an option for companies facing this big data skills gap — the emergence of application performance management (APM) for Big Data. Although commonplace in other areas of IT Operations, APM has only recently been introduced to support the big data ecosystem.  An APM platform allows users to optimize, troubleshoot, and analyze the performance of big data applications that are running in highly distributed, multi-tenant environments. With the help of APM, both business analysts and IT operations teams can mitigate big data app challenges without the need for special skills or expert knowledge.

Looking at the benefits

An APM platform monitors the entire big data stack and collects information at every layer, providing comprehensive management and optimization of the full deployment. Overall, APM allows users to do the following:

Determine root cause of big data app failures: When a big data app fails, it’s not easy to find out why, and to do so, you essentially need PhD-level expertise. APM leverages historical app data (looking at examples of both success and failure for a given app) to determine why it broke down. The cause can be a resource contention issue, bad configuration, poor partitioning, flawed code, version issues, and other reasons. APM takes out the guesswork and delivers quick answers.

Optimize app performance: App failure isn’t always the biggest challenge in a deployment, sometimes big data apps run but simply don’t perform well. Like failures, these performance issues have many causes. APM provides detailed insight into how an app is being used, its resource consumption and its dependencies with other apps. This insight makes it possible to drastically improve speed and reliability.

Improve resource utilization: Big data apps often run on the same clusters and share the same compute resources. APM highlights how many resources each app is using, enabling operations to reconfigure apps that are using too many resources and slowing down other apps. This ensures that the highest priority apps can get the resources they need to perform at a required level.

Detailed analytics for every big data app: APM gives users an exhaustive view of how all big data apps are performing. This includes comprehensive metrics on users, usage, storage, CPU and queue data. Some big data apps have their individual analytics features, but these analytics capabilities are typically very bare bones and don’t provide many insights.

Even more powerful with automation

These capabilities are even more powerful when they’re combined with automation, which is where AIOPs comes in. AIOps is an emerging capability that combines big data, artificial intelligence, and machine learning functionality to enhance and replace many IT operations processes. Gartner predicts that by next year, 25 percent of global enterprises will have implemented an AIOps platform to support two or more major IT operations. AIOps is being incorporated as part of some APM platforms to automatically diagnose app failures and then automatically fix them.

The AIOps technology combines the big data stack’s operation metrics with recommendations and actions to deliver the desired business outcome. In short, AIOps automates the most important features of APM – diagnosing and fixing failing apps.

With big data proving that it’s here to stay, more people will be driven to careers as experts in the technology. However, the workforce will never be great enough to meet the massive demand for big data skills, and that demand will only increase as AI, ML and other new uses for big data continue to see rising adoption. APM can help organizations bridge the big data skills gap.