Part 7 — Inference Performance and Economics

Cost per successful result

The economics·Evaluation·7 min read

The metric that unites the two halves of this book. Cost per request counts every answer equally, including the wrong ones — but you only get value from correct answers. Cost per successful result divides cost by accuracy, and it can completely reverse which system is 'cheaper'.

Here economics and evaluation finally meet, and the result overturns a lot of naive cost reasoning. Cost per request counts every answer the same — but a wrong answer has no value, or negative value once someone has to catch and redo it. What you actually pay for is a correct answer, so the metric that matters is cost per successful result = cost per request ÷ accuracy. That single division ties the whole book together, and it can flip the ranking: the cheapest system per request is frequently the most expensive per useful result.

What you will understand by the end

  • Why cost per request is the wrong economic metric on its own.
  • The formula: cost per success = cost per request ÷ success rate.
  • How a cheap-but-inaccurate system can cost more per useful answer than an expensive-but-accurate one.
  • Why this is where performance and evaluation become one decision.

Wrong answers aren't free — they're worse than free

A request that produces a wrong answer still cost you tokens or GPU-time to generate — and then it costs again: a human catches and corrects it, a user retries, a downstream process fails, or worst, the error ships. So counting cost per request treats a wrong answer as an equal to a right one, which is exactly backwards. The economically meaningful denominator is not requests, it's successes — the answers that were actually correct and usable.

Key idea

Cost per successful result = cost per request ÷ success rate. You pay for every request but get value only from the correct ones, so dividing by accuracy is what converts a raw cost into the cost of a useful outcome. A system's per-request cost is only half the economics; its accuracy is the other half, and they multiply.

The ranking can reverse

This is the punchline: the cheapest system per request is often the most expensive per success. Consider two systems on the same task:

   System A (cheap, inaccurate)    System B (pricey, accurate)
   cost/request:  $0.001           cost/request:  $0.010   (10× more)
   accuracy:      45%              accuracy:      95%
   ──────────────────────────      ──────────────────────────
   cost/SUCCESS = 0.001 / 0.45     cost/SUCCESS = 0.010 / 0.95
              ≈ $0.0022                       ≈ $0.0105

By cost per request, A looks 10× cheaper. By cost per success, A is still cheaper here — but the gap collapsed from 10× to ~5×, and it keeps collapsing as accuracy diverges. Push A to 20% accuracy and B to 98%, and B wins outright. And per-request cost ignores the downstream cost of wrong answers entirely (the human who fixes them, the incident) — fold that in and the accurate system wins far sooner. Low accuracy is a cost multiplier hiding in plain sight.

Watch out

Choosing a model or system on cost per request (or worse, on token price) systematically over-values cheap, inaccurate systems, because it never charges them for being wrong. A model that's half the price but far less accurate can be the more expensive choice per useful answer — and much more so once you count the human and downstream cost of cleaning up its errors. Always divide by accuracy before comparing costs.

Where the two halves of the book meet

Every earlier part pays off here. You need cost per request (this part) and a trustworthy accuracy (the evaluation half) to compute cost per success — and neither alone answers the question "which system should we run?" A guardrail that raises accuracy improves cost per success even if it adds a little cost per request; a cheaper model that drops accuracy can worsen it. This is the metric on which the hosted-vs-self-hosted and model-comparison decisions should actually turn.

Observed evidence

This project is a live illustration: the six systems ranged from 45.5% to 95.5% accuracy at different per-request costs (cheap self-hosted open models vs pricier hosted frontier models). Cost per request alone would crown the cheap 45.5% system; cost per success — dividing by accuracy, and remembering every wrong intent needs catching — tells a very different story, and is exactly why the project pairs a correction policy on a cheap model against a pricey accurate one. The whole comparison only makes economic sense per success. Accuracy and system alongside cost — the cost-per-success picture →

Mental model

Cost per successful result = cost per request ÷ success rate. You pay per request but profit only from correct answers, so accuracy is a cost multiplier: the cheapest system per request is often the most expensive per useful result, especially once you count the downstream cost of fixing wrong answers. This is where the performance half and the evaluation half of the book become a single decision.

Common mistakes

  • Comparing systems on cost per request. It over-values cheap, inaccurate systems by never charging them for being wrong.
  • Ignoring the downstream cost of errors. Wrong answers cost again — to catch, redo, or recover from — beyond their generation cost.
  • Optimising cost without re-checking accuracy. A cheaper model that drops accuracy can raise cost per success.
  • Treating accuracy and cost as separate decisions. They multiply into cost per success; you can't choose a system on one alone.

Practical guidance

  • Compare systems on cost per successful result (cost per request ÷ accuracy), not cost per request or token price.
  • Fold in the downstream cost of wrong answers (human correction, retries, incidents) — it shifts the economics toward accuracy.
  • Value an accuracy-raising guardrail or better model by its effect on cost per success, not just per-request cost.
  • Make the hosted-vs-self-hosted and model decisions on this metric — it's the one that reflects real value.

Summary

  • Cost per successful result = cost per request ÷ success rate — you pay per request but profit only from correct answers.
  • Accuracy is a cost multiplier: the cheapest-per-request system is frequently the most expensive per success, more so once you count the downstream cost of errors.
  • It's the metric where performance and evaluation become one decision — needing both a cost and a trustworthy accuracy.
  • Choose systems, models, and guardrails on cost per success, not cost per request.

Knowledge check

Model A costs $0.002/request at 50% accuracy; Model B costs $0.008/request at 96% accuracy. Which is cheaper per useful answer, and what extra factor could change the conclusion?

Cost per success = cost/request ÷ accuracy. A: 0.002 / 0.50 = $0.004 per correct answer. B: 0.008 / 0.96 ≈ $0.0083 per correct answer. So by cost per success A is still cheaper (~2×, versus 4× on cost per request — the accuracy gap already halved A's apparent advantage). The extra factor that could flip it is the downstream cost of wrong answers: A is wrong 50% of the time and B only 4%, so if each wrong answer costs meaningfully to catch, redo, or recover from (human review, retries, incidents), A's effective cost per useful outcome climbs sharply and can exceed B's. Once error-handling cost is included, the accurate model often wins.

Your team wants to cut costs by switching to a cheaper model. What must you measure before deciding, and why is token price alone misleading?

You must measure the cheaper model's accuracy on your task and compute cost per successful result (cost per request ÷ accuracy), including the downstream cost of the extra wrong answers it produces. Token price alone is misleading because it only captures cost per request and never charges the model for being wrong — a cheaper model that's meaningfully less accurate generates more wrong answers, each of which cost to produce and cost again to catch and redo, so it can easily be more expensive per useful answer despite the lower sticker price. The switch is only a real saving if cost per success (with error-handling folded in) actually drops.

Related chapters