Part 7 — Inference Performance and Economics

Throughput and concurrency

Throughput and load·Serving·6 min read

Throughput is what your cost-per-token depends on, and it rises with concurrency — until it doesn't. This chapter is about the throughput-vs-latency relationship, the knee where more load stops buying more work, and why finding that knee is the central operating decision of a serving system.

Latency is what one user feels; throughput is what your economics depend on — the total work the system does per unit time, usually tokens/sec or requests/sec. It rises as you serve more requests concurrently, because batching reuses each expensive weight read across the batch. But it doesn't rise forever: at some point more concurrency stops buying more throughput and only buys latency. That point — the knee — is the single most important number in operating a serving system, and this chapter is about finding and respecting it.

What you will understand by the end

  • Why throughput rises with concurrency, and why it eventually stops.
  • The knee: where added load stops buying work and starts buying only latency.
  • The throughput-vs-latency trade, and why you operate just below the knee.
  • Why the knee is workload-specific and must be measured.

Throughput rises with concurrency — up to a point

Recall from batching that decode is memory-bandwidth-bound, and the weight read is shared across everyone in the batch — so serving more requests concurrently converts idle compute into tokens almost for free. Throughput climbs with concurrency. But the resources are finite: eventually the GPU's compute or its KV-cache memory saturates, and adding more concurrent requests no longer produces more tokens per second — it just makes each request wait.

   throughput
   (tokens/sec)        ___________ ← plateau: saturated, no more work per second
              │      /
              │    /  ◀── the KNEE
              │  /
              │/
              └───────────────────────▶ concurrency (offered load)
              past the knee: throughput flat, but latency climbs steeply
Key idea

Throughput scales with concurrency until a knee, where a resource (compute or KV memory) saturates. Before the knee, more load buys more work at little latency cost; after it, more load buys no extra throughput and only piles on latency. The knee is the system's real capacity, and operating a serving system is largely about knowing where it is.

The trade, and where to operate

Throughput and latency pull against each other, and the knee is where the trade turns bad:

  • Below the knee — spare capacity; adding concurrency raises throughput while latency stays acceptable. This is the efficient region.
  • At the knee — maximum useful throughput; the system is fully utilised and latency is still bounded.
  • Past the knee — throughput flat, queue growing, tail latency blowing out. You're paying in latency for throughput you're not getting.

So the operating point is just below the knee: enough load to use the hardware efficiently (good cost-per-token), not so much that latency violates your SLO. Push past it and you degrade every user's experience for zero throughput gain.

Watch out

"More concurrency = more throughput" is only true up to the knee. Cranking concurrency past it is a classic mistake: the throughput graph flatlines while p99 latency explodes, so you've made every request slower and served no more work. If raising load isn't raising tokens/sec, you're past the knee — the fix is more capacity or less load, not more concurrency.

The knee is workload-specific — measure it

Where the knee sits depends on the model, the hardware, and the workload's shape (prompt vs output lengths, which decide whether compute or KV memory saturates first). You cannot read it off a spec sheet; you find it by load testing — ramping concurrency and watching where throughput plateaus and latency turns up.

Observed evidence

This project's serving sweep found the knee empirically on an L4: throughput climbed with concurrency to a distinct plateau, after which added load only grew the queue and tail latency. The knee was compute-bound — the KV pool sat near 1.4% full, so memory wasn't the limit — which a spec sheet couldn't have told you. Finding that knee by measurement, and serving below it, is the whole operating decision. See the throughput knee measured on real hardware →

Mental model

Throughput rises with concurrency because batching shares the weight read — until a resource (compute or KV memory) saturates at the knee. Before the knee, load buys work; after it, load buys only latency. Operate just below the knee — efficient hardware use within your latency SLO. The knee is workload-specific and must be measured, not read off a spec sheet.

Common mistakes

  • Assuming more concurrency always helps. Past the knee it adds latency and zero throughput.
  • Guessing the knee from specs. It depends on model, hardware, and workload shape; measure it.
  • Operating past the knee for "utilisation." A saturated system with a huge queue serves no more work and fails its SLO.
  • Ignoring which resource saturates. Compute-bound vs KV-bound knees have different fixes (bigger card vs more memory/attention variant).

Practical guidance

  • Measure the knee by ramping concurrency and watching where tokens/sec plateaus and tail latency turns up.
  • Operate just below the knee — maximum efficient throughput within your latency SLO.
  • Identify which resource saturates (compute vs KV memory) so you scale the right thing.
  • Use a max-concurrency / batch cap to keep the system from being pushed past the knee under load spikes.

Summary

  • Throughput (tokens/sec, req/sec) rises with concurrency because batching shares the weight read — until the knee.
  • Before the knee, load buys work; past it, load buys only latency while throughput plateaus.
  • Operate just below the knee — efficient utilisation within your latency SLO.
  • The knee is workload-specific and must be measured, as this project did on its L4.

Knowledge check

You raise concurrency from 40 to 80 and tokens/sec doesn't increase, but p99 latency doubles. What happened, and what should you do?

You pushed the system past its knee. At 40 you were at or near maximum useful throughput; going to 80 added offered load a saturated resource (compute or KV memory) can't turn into more tokens/sec, so throughput plateaued — while the excess requests queue, blowing out p99. More concurrency bought only latency. The fix is not more concurrency: either add capacity (scale the saturated resource — compute vs KV memory determines which) or shed/cap load to stay just below the knee, where throughput is maximal and latency is still within SLO.

Why can't you determine your serving system's throughput knee from the GPU spec sheet?

Because the knee depends on the interaction of model, hardware, and workload shape — specifically whether compute or KV-cache memory saturates first, which is set by your prompt and output lengths, batch sizes, and attention variant, none of which the spec sheet knows. A prompt-heavy workload hits a compute-bound knee; a long-context, high-concurrency one may hit a KV-memory knee — on the same card. The spec sheet gives peak FLOPS, bandwidth, and VRAM, but where your traffic plateaus is an emergent property you can only find by load testing the actual system with representative traffic.

Related chapters