There’s an interesting analogy I like to draw between the state of enterprise data today and where containers were about ten years ago.
Early in the container revolution, organizations started realizing they were supporting an entire infrastructure layer that was exponentially proliferating under them. The solution wasn’t to stop shipping containers or build better containers – it was to build a control plane above them. Kubernetes abstracted orchestration away from infrastructure, and overnight complex tasks became manageable.
Telemetry has hit the same breakpoint. Data flowing through modern enterprise environments has massively outpaced the tooling and architectures designed to manage it – and the gap is only growing. I’m talking to engineers who aren’t questioning if they need telemetry pipelines. They know they do. The question is how quickly they can spin them up.
See also: The RAG Pipeline Nobody Told You Was Unnecessary
Two Waves of Telemetry Growth
The first wave came with cloud-native transformation. Microservices architectures, distributed deployments, and container sprawl all generated orders of magnitude more telemetry than the monolithic systems they replaced.
The second wave is AI. Every new AI use case, be it AIOps, security analytics, observability tooling, or compliance monitoring, generates its own telemetry streams and demands its own data inputs. Data engineers are managing many times the volume they handled five years ago, and the infrastructure assumptions baked into their current architectures weren’t designed for this reality.
What Happens Without a Pipeline Layer
One of our customers operates security services at significant scale. Their telemetry volumes were climbing steadily – within the range their SIEM could absorb. Then they hit a wall.
The data kept coming. Their SIEM couldn’t keep up. The instinct was to buy a bigger one.
The actual solution was a pipeline layer. Consider web server logs: on a typical web server, the overwhelming majority of entries are 2xx-status messages confirming normal operation. They carry no detection value. Apply filtering at the pipeline layer – condense normal traffic to a summary, surface errors that warrant attention – and you’ve already made a significant dent in your volume problem.
Repeat that logic across Windows events, application logs, and infrastructure telemetry, and you can reduce total data volumes by 40 percent or more without losing meaningful visibility. The critical word is “before.” Filtering only works if it happens before the data reaches your SIEM or observability platform.
A Binary Strategic Choice
The historical approach to telemetry was simple: buy a platform, use the collection tools it came with. The tools weren’t always great, but they were bundled, and they worked well enough. That calculus no longer holds.
AI adoption is accelerating the introduction of new platforms, new data sources, and new analytical capabilities at a rate that makes vendor-locked telemetry architectures operationally untenable. When your data collection mechanism is tightly coupled to a single vendor, evaluating a new platform isn’t a week of work; more likely it’s a six-month refactoring project.
Owning your pipeline changes that equation completely. If you’ve decoupled your data plane from any individual vendor’s infrastructure, routing traffic to a new platform becomes a configuration change. You evaluate, you learn, you decide, in days, not months.
This is the choice now facing every organization building a real-time data strategy: control your pipeline or permanently cede it to vendors. As AI adoption continues and data volumes expand, the cost of the latter will only compound.
What a Production-Grade Pipeline Requires
The good news is that the foundational layer already exists. OpenTelemetry has established itself as the industry standard for telemetry ingestion, giving organizations a vendor-neutral instrumentation and transport format to build on. A well-constructed pipeline built on top of that provides collection, enrichment, data security, volume reduction, and intelligent routing.
What separates a robust pipeline from a minimal one is what comes on top: real-time visibility into data in motion, the ability to modify routing without re-instrumenting your sources, and the resilience to operate that infrastructure reliably at scale.
The Engineering Reality
Building a telemetry pipeline is a serious upfront investment. Standardizing on an internal schema, defining what that schema should look like, and getting application and infrastructure teams to adopt it takes time and organizational alignment. That cost is real and shouldn’t be understated.
But once the foundation exists, the economics shift. You can scale a standardized pipeline to the full organization. You can enforce schema compliance with tooling rather than process overhead. And your data engineers can redirect their attention from managing ingestion complexity to solving harder, more valuable problems.
A consistent pattern emerges when I talk to observability and security teams: ask how many platforms they’re ingesting from, and roughly half the room says “all of them” – a dozen different vendors, a dozen different ingestion mechanisms, no common data model. A properly constructed telemetry pipeline resolves that structural problem. Data that was previously a management liability, something to be contained, becomes a clean, normalized organizational asset.
Why AI Raises the Stakes on Pipeline Architecture
AI introduces new categories of telemetry requirements: instrumenting and collecting for AI workloads, capturing prompt and response data, evaluating model effectiveness, and enforcing compliance around what data is being transmitted to which models. In regulated environments, particularly in the EU, the stakes around data residency and AI data handling are significant and increasing.
A normalized, vendor-agnostic telemetry pipeline built on OpenTelemetry addresses these requirements as a byproduct of the architecture rather than a separate compliance layer. If you already have a unified pipeline that governs where data flows, you can enforce data residency rules globally – for AI workloads, security, observability – from a single control point. If your telemetry is still fragmented across vendors with incompatible ingestion mechanisms, applying those same rules means retrofitting controls across a dozen systems with no common architecture.
Compliance requirements aren’t vendor-aware. If a regulation says certain data can’t cross a geographic boundary, you need one authoritative place to enforce that rule. A telemetry pipeline is that place, if you’ve built one.
The organizations that make this investment now will have a structural advantage in cost efficiency, operational agility, and AI readiness for the foreseeable future.
