Every powerful system society has learned to trust has independent oversight. Pilots have air traffic control. Surgeons have timeouts and second attendings. Nuclear reactors have independent monitoring systems with the authority to scram the core. Banks have risk officers who do not report to the trading desk. We did not build any of these supervisory systems first. We built them, in every case, after we lost things we could not afford to lose.
Large language models do not have one.
They write contracts, draft medical assessments, and generate code that runs in production. They increasingly touch consequential decisions in defense, finance, and healthcare. And the losses have started arriving on schedule. A researcher at HEC Paris now maintains a database of court cases in which AI-generated fabrications, invented citations, fake quotes, and nonexistent precedents were filed with real courts. It passed 1,700 cases this spring, growing at a pace he describes as ten different courts in a single day. The first sanction, in 2023, was $5,000. This April, an Oregon federal court imposed $110,000, and Nebraska issued the first bar suspension.
A tribunal has already ordered an airline to honor a refund policy its chatbot invented on the spot. These are the visible failures, the ones that happened to occur in front of a judge. The interesting question is what the failure rate looks like everywhere no judge is watching.
When these systems are checked at all, they are checked by other large language models with the same architecture and the same blind spots. That is worth calling what it is: a structural, architectural defect, no matter what the regulations eventually say.
See also: Domain-Specific LLMs: How to Make AI Useful for Your Business
The Current Answer is Not an Answer
The industry’s response to LLM unreliability has been to make LLMs better at supervising themselves. Reinforcement learning from human feedback makes them more helpful. Chain-of-thought lets them show their work. Constitutional AI gives them a written set of values. Ensemble methods take votes among models. Each of these is genuinely valuable. None of them is supervision in the sense the word carries in any other domain.
A pilot does not certify her own flight plan. A nuclear plant does not declare itself safe. The supervisor is, by design, structurally separated from the system it oversees, with different inputs and different failure modes. An LLM checking another LLM’s output is, at best, like asking a pilot to check his own flight plan.
The record bears this out with almost comic precision. In the first sanctioned case, the lawyer grew suspicious of the citations ChatGPT had given him, so he asked ChatGPT whether the cases were real. It confirmed its own fabrications, complete with reassurances. Researchers have since found the same thing in controlled settings: asked to review their own reasoning without outside information, models frequently reaffirm their errors, and performance can degrade rather than improve. Stanford researchers testing legal queries measured hallucination rates of 69 to 88 percent for general-purpose models, and purpose-built legal AI tools, marketed on their reliability, still produced false information in roughly one in six queries or more.
The deeper problem is that an LLM’s failure modes are not reliably detectable by another system with the same architecture. Hallucination, sycophancy, prompt injection, mode collapse, confident wrongness: these are emergent properties of the design rather than bugs awaiting a patch. A supervisor that shares the architecture shares the failure modes. It will silently agree on the wrong answer, and we will learn that it did only after something has already gone wrong.
What a Real Supervisor Would Need
Asked in good faith what an independent supervisor for a language model would require, I think the answer has five parts.
- Architectural independence. It cannot be another LLM, and the reason is foundational. A language model is a probabilistic system. It produces the statistically likely next word given its training, which means its errors are systematic byproducts of what was likely in the data rather than random noise. Two probabilistic models trained on largely the same internet will make correlated mistakes, so when they agree, their agreement measures shared statistics rather than truth. Ten LLMs can vote unanimously for the same wrong answer, and in that vote the confidence goes up while the accuracy does not. A supervisor must rest on a different foundation, one that evaluates claims against a structure it has independently built rather than sampling from the same distribution, so that the two systems fail differently and cannot silently agree. Aviation requires that primary and backup hydraulics share no common failure mode; nuclear safety requires that monitoring instruments not draw from the same power supply as the systems they monitor. Independence is the entire point.
- A persistent, independent ground truth. A supervisor needs its own memory of what has been established as true, separate from whatever the LLM is currently asserting. Without it, the supervisor cannot distinguish a fluent, confident falsehood from a fluent, confident fact. It is just another voice in the room. The value of a supervisor is that it brings a different ledger to the conversation.
- The authority to withhold. The supervisor must be able to refuse to pass an answer through when its checks fail, rather than auto-correcting by re-querying the same model. A second opinion from the same source is no opinion at all. The capability that makes it a supervisor rather than an editor is the ability to say no and make it stick.
- Continuous learning, deployed where the work happens. A supervisor that must be retrained whenever the world changes is permanently six months behind. One that lives in a distant data center adds latency and an external dependency that defeats the purpose. Real decisions happen on real devices in real time. The supervisor needs to be there, and it needs to keep learning there.
- Auditability. When the supervisor accepts or rejects an output, it must be able to say why, in a chain of reasoning a human investigator can follow after a failure. We have a name for systems that make consequential decisions and cannot explain themselves, and we do not let them fly airplanes. We should not let them oversee artificial intelligence.
Notice what is absent from this list: scale. More parameters and more training data are LLM virtues. A supervisor is valuable because it is different, and difference cannot be purchased with size.
Why Not Just Use the Supervisor?
A fair question follows: if the supervisor is reliable enough to overrule the language model, why keep the language model? Because they are different kinds of systems doing different jobs.
The language model is generative. It drafts the contract, summarizes the deposition, writes the working code, and explains quantum mechanics to a ten-year-old. Breadth of fluent expression is its competence and its purpose. The supervisor evaluates rather than generates. Its competence is depth of judgment on a narrow set of questions. Is this claim grounded? Does it match what has been established? Should it pass, or be held?
Air traffic control does not fly the plane. The risk officer does not place the trade. In every supervised domain, the supervisor and the operator are complements, and neither can do the other’s job. You cannot ask the controller to land the plane better than the pilot. You can ask her to keep the pilot off the wrong runway. A supervisor that could also do the language model’s job would, by definition, share its architecture, and therefore its failure modes. The difference is what makes it useful, and the difference is also what makes it narrow. The future is language models with supervisors, each doing what the other cannot.
See also: The Hard Part of Agentic AI Is Authority, Not Accuracy
Why We Don’t Have This Yet
The economic incentives point the other way. Companies that build foundation models are paid for fluency. A real supervisor would, by design, sometimes refuse to answer, and that reads as a product regression even when it is a safety win. The quarterly earnings call does not reward safety wins.
The regulators haven’t arrived either. The FDA does not yet require an independent monitoring layer for an LLM giving medical advice, and the SEC does not yet require one for an LLM touching securities. The Department of Defense is starting to ask the question. Most others have not.
And the technical community is inside the paradigm, convinced that any problem with LLMs can be solved by better LLMs. Every supervised domain passed through this stage and paid tuition. In 1956 two airliners collided over the Grand Canyon in uncontrolled airspace, killing 128 people, and within two years Congress had created the FAA. When the World Health Organization piloted a simple surgical checklist across eight hospitals, deaths fell by nearly half, and this from a tool many surgeons had insisted they did not need. Nuclear power adopted true defense in depth only after Three Mile Island. The lesson every domain eventually learns is the same: a powerful system’s failures are not fixed by making the system more powerful. They are fixed by giving it a sober second mind.
What Comes Next
The next phase of useful AI is the layer that makes scale safe to use. Working out what supervision actually means for systems that produce language is the most important open problem in applied AI today, and the foundation model labs are the least likely to solve it, for the same reason Boeing or the airlines were never going to invent the FAA. The history of supervision in every domain is a history of independence, of institutions structurally separated from the systems they oversee. There is a reason we do not let powerful systems certify themselves.
The frontier in AI right now is to create a system that sits above the models, rather than inside them. The companies and researchers who take that seriously will define the next decade, and will do for AI what air traffic control did for aviation: turn a powerful but unreliable technology into one that society can actually trust.
