Vercel recently published a piece on inference theft, where attackers wrap your AI endpoint in an OpenAI-compatible adapter, fan calls through residential proxies, and resell your paid inference at a discount. What jumped out at me was the cost asymmetry: inference is the most expensive resource, and verification is one of the cheapest protections you can put in front of it. When the gap is that wide, you gate every request, because the gate pays for itself.
The same logic can be applied to address an even more common source of excess token usage: employees who are using the company’s AI, casually, constantly, and without a thought for the bill. Almost everyone providing an internal assistant has this issue, whether they’ve noticed it or not.
The intent isn’t malicious at all: employees are asking the assistant to plan a weekend in Hawaii, rewrite a personal email, or draft a limerick for a retirement card. One person doing this is noise. Fifty thousand people doing it a few times a day, every working day, is a real line item – and at frontier-model prices, the costs reach into the millions before anyone files a ticket.
The instinct here is governance: AI should remain focused on the tasks it was deployed for, and that is a policy decision before it is a billing one. The surprising part lies in the economics. Screening prompts is almost free compared with generating answers because most of the cost comes from producing long outputs. As a result, blocking even a tiny fraction of off-task requests can cover the cost of the screening itself. The rest of this piece works through the arithmetic to show when this kind of governance makes sense, and when it doesn’t.
Task moderation
Task moderation means looking at a prompt before it reaches the expensive model, deciding whether it’s on-task, and stopping prompts that aren’t. The mechanics matter for the economics: you pay to screen every prompt, but you only avoid generation costs on the prompts you block. Screening is pure overhead on allowed traffic. The savings come entirely from the blocked slice. The whole question boils down to this: what share of traffic would you need to block for screening to pay for itself?
No need to guess – let’s model it.
The proof
Let:
- M be the average cost of serving one query on your main model, and
- S the average cost of screening one query.
You screen every query but serve only the ones that pass – so you spend S on every query and save M only when a query is blocked. The savings overtake the spend exactly when the blocked share reaches S/M. That ratio is the answer: the break-even rate – the fraction of traffic you’d have to prevent to cover the cost of screening. Call the share you actually block b, and screening comes out ahead when b is greater than the block rate.
b > S / M
That’s the simple version of the whole argument, but it’s worth expanding S / M into the three things that actually move it.
Let:
- k be the model’s output-to-input price ratio,
- r be the output-to-input token ratio for a typical query, and
- c be your screening input price relative to the model’s input price
Then the break-even block rate is:
b* = c / (1 + k·r)
The market sets k, your workload sets r, and the screening vendor sets c.
It helps to read that equation to build some intuition for what it says. First, the denominator is where the leverage lives. k, the output-to-input price gap, is 4-5 on current frontier pricing; r, the number of output tokens per input token, runs large for chat – at least 10:1, often more than 25:1. Multiply them, and you’re in the dozens before doing anything clever. The numerator c has a natural ceiling: there’s little reason to pay more per token to screen than you already pay for input, so c = 1 is the worst case, and a good judge model will usually run at a c well below 1. So it’s a small number over a big one.
Now, we can take the least favorable plausible case – c = 1, k of 4 (the low end), a merely modest r of 10 – and you get 1 / (1+40), about 2.4%. Using more realistic values only improves the argument this equation makes: a cheaper judge (c below 1), the more common k of 5, or a chat-typical r north of 20. Halve the screening price, and you’re near 1%; let r reach 25, and you’re at 0.8% (the next section works this out in dollars). This is good news, because it tells us that even the worst case asks you to block only a sliver of off-task traffic to have governance pay for itself.
Let’s talk brass tacks
Assume we have a single off-task chat query – a twenty-token prompt that will generate a five-hundred-token answer – priced on a mid-tier frontier model at $3 per million input tokens and $15 per million output.
To tie this back to the expanded equation above (and continue to build our intuition), that means:
- k = 5 – output is priced at five times input ($3 vs $15)
- r = 25 – five hundred output tokens for twenty input
- c = 0.5 – assumes screening costs half of inputs, quite reasonable
If we plug all this in, we find that serving this query will cost about 0.76 cents (set M to this): a hundredth of a cent for the input, three-quarters of a cent for the output. Notice that the output cost is more than ninety-nine percent of the bill – this plays a large role in the efficiency of screening. Moving on, screening the input costs about 0.003 cents (set S to this): the screening model reads the same twenty-token prompt, and with no answer to generate, there’s no output-token cost. Now we just do simple division: S / M is about 0.39 percent.
Now let’s suppose we receive 1000 of these requests. Screening is unconditional, so you pay 0.003 cents on each of the thousand prompts – about three cents of tax in total. You earn it back only on the queries you block, at 0.76 cents each. Block two percent of the thousand – twenty queries – and you recover 15.2 cents against three cents of tax: 12.2 cents saved per thousand queries. Multiply this amount by an enterprise’s daily volume, and it’s a real line item.
Where it works, and where it doesn’t
Whether this saves money depends on the shape of your workload.
The model assumes the traffic is shaped like AI chat, with a short prompt and a long answer. This means r will inevitably be large, which in turn will make the denominator large and the break-even point favorable. But it is important to remember that r is set by your workload and there are workloads where r won’t be quite as large. For example, retrieval and summarization may sit at the opposite end: you push thousands of tokens of context in and get a few hundred back, so r drops well under one, the denominator (1 + k·r) collapses toward 1, and break-even climbs toward one hundred percent. That’s the equation’s polite way of saying “don’t expect a refund,” and the equation will tell you whether it makes sense for your use case.
That said, the input-heavy case might be less fatal than it first appears. Three design choices can change the calculus.
The first fix is to screen the intent, not the payload. This is the important one, and it’s where a naive integration fails. If you assume you have to send the screening system everything the model sees (or will see) – the user’s prompt, the system prompt, and every retrieved document plus the tool schemas – your screening cost will of course balloon in lockstep with the model’s input.
But it’s worth asking: “Does the screening classifier actually need all of that input?”
Whether a request is on-task is likely to be determinable from just the user’s prompt. The context your pipeline assembled around the prompt probably doesn’t add anything that would help to determine intent. So instead, let’s route just the user’s prompt. This way, the r that governs your screening cost is computed against a small input, not the inflated one. The same move rescues the other input-heavy cases – long document analysis, agents with large tool schemas, conversations dragging a long history – because in nearly all of them the expensive part of the input isn’t the part that reveals intent.
There’s even a small bonus that falls out of the sequencing: Because the screening happens before retrieval, blocking an off-task request saves on more than the generated output tokens; it saves on the inflated input tokens that would have been assembled and processed as well. Viewed this way, the context bloat that threatened the economic viability of screening actually becomes extra savings you can put in your pocket.
The second is that what you do block works harder than you’d expect. Off-task personal use skews toward the long, indulgent tail – creative writing, open-ended planning, ‘keep going.’ These aren’t average queries. The average blocked query is likely more expensive to fulfill than the average query overall, so the savings per block beat the mean.
The third is that screening stops the cascade before it starts. For an agentic assistant, a single off-task prompt costs more than one generation – it spawns retrievals, tool calls, and multi-turn loops. Screening at the front door kills the whole tree. That’s where the very large savings could live.
A note on screening price
There are many ways to employ this kind of task moderation. You can use a vendor, you could have your usual frontier model act as its own judge, or you could set up an LLM of your own to do the same. You could even train a binary classifier if you’re willing/able to accurately label the data, your policy isn’t likely to change very often, and you don’t need deep explainability. As such, the cost of screening tokens relative to input tokens can vary quite a bit. But it’s unlikely screening tokens would ever cost more than input tokens. c = 1 is truly the worst case.
This also shows that the price per screening token really isn’t load-bearing. Pay the same price as you currently do per input token, and the math says that you’ll break even with a chat-shaped workload if you block 0.8% of requests (1 in every 125). The economics are a property of the input/output price gap, as well as the disparity in the number of input vs. output tokens – reducing the cost of screening tokens is just a cherry on top. Still, you are likely to find that the cost per token to do screening is quite a bit less expensive than the cost to run your AI generally.
The bottom line
Is task moderation a cost-reduction tool? In quite a few cases, yes.
It’s a governance control always, and in the regime internal assistants actually live in – short prompts, long generated answers – it pays for itself. And if, instead, your workload is dominated by input-heavy retrieval and your off-task rate is low, you can always screen for policy reasons and treat any savings as a rounding error (or get clever about the sequencing of your governance controls).
