Load testing

Everything in this part so far — the knee, the tail, saturation — is a property you must measure, because it emerges from the interaction of model, hardware, and workload and appears on no spec sheet. Load testing is how you measure it: drive the system with controlled, realistic traffic, ramp the load, and watch where throughput plateaus and latency turns up. Done honestly it turns capacity planning from guesswork into a number; done carelessly it produces confident, wrong answers just like a bad eval.

What you will understand by the end

• Why the knee and tail can only be found empirically, by load testing.
• How to run a load test: ramp concurrency, measure the right metrics.
• Why the test traffic must be realistic, or the numbers mislead.
• What the result gives you: the knee, and the inputs to capacity planning.

Why you must measure

The throughput knee depends on whether compute or KV memory saturates first, which depends on your prompt and output lengths, batch sizes, and model — an emergent property no spec sheet predicts. So you find it the only way you can: push real load at the real system and observe. This is the performance twin of the book's evaluation lesson — theory says what should bind; a benchmark says what does, and the two don't always agree.

How to run one

The core method is a concurrency ramp:

• Ramp offered load — start low and increase concurrency (or request rate) in steps, holding at each level long enough to reach steady state.
• Measure throughput and latency together at each level — tokens/sec and req/sec alongside p50/p95/p99 of TTFT, ITL, and end-to-end, plus queue depth and resource utilisation (compute, KV pool).
• Find the knee — the load where throughput plateaus and tail latency turns up. That's your capacity.
• Note the binding resource — which of compute or KV memory saturated first tells you what to scale.

   ramp:  10 →  20 →  40 →  60 →  80 → 100 concurrency
          │     │     │     │     │     └ throughput flat, p99 exploding  ← past knee
          │     │     │     │     └ knee: max throughput, p99 still ok
          └─────┴─────┴─────┴ throughput rising, latency stable          ← below knee

Make the traffic realistic

A load test measures the workload you give it, so an unrealistic workload gives an unrealistic knee. The traffic must match production in the dimensions that move the knee:

• Prompt and output lengths — these decide whether the knee is compute- or memory-bound; testing with short prompts when production sends long ones measures a different system.
• Arrival pattern — steady vs bursty; bursts saturate earlier than the average load implies, so a smooth test overstates capacity.
• The mix — the real distribution of request types, including the expensive tail.

Mental model

Load testing is the experiment that finds the knee, the tail, and the saturation point — which no spec sheet predicts. Ramp offered load in steps, measure throughput and p50/p95/p99 together plus queue depth and the binding resource, and read off the knee where throughput plateaus and tail latency turns up. Use representative traffic (lengths, burstiness, mix), or you measure an easier system than production.

Common mistakes

• Guessing capacity from specs. The knee is emergent and workload-specific; only a load test finds it.
• Unrealistic test traffic. Short uniform prompts and steady arrivals report a too-high knee that production won't hit.
• Measuring throughput without latency (or vice versa). You need both together to locate the knee — the point where one plateaus and the other turns up.
• Not holding each load level to steady state. Transient measurements misread the knee; let queues settle before recording.

Practical guidance

• Run a concurrency ramp, holding each level to steady state, and record **throughput + p50/p95/p99
  • queue depth + resource utilisation** together.
• Use representative traffic — production-like prompt/output lengths, arrival burstiness, and request mix — so the knee is real.
• Read off the knee (throughput plateau + tail turn-up) and the binding resource, and feed both into capacity planning.
• Re-load-test after any change to model, hardware, or workload shape — the knee moves with all three.

Summary

• The knee, tail, and saturation point are measured, not derived; load testing is the experiment that finds them.
• Run a concurrency ramp, measuring throughput and tail latency together plus queue depth and the binding resource, to locate the knee.
• Test with representative traffic (lengths, burstiness, mix) or you measure an easier system than production and overstate capacity.
• The result feeds capacity planning and SLOs — this project's serving sweep is the method in practice.

Knowledge check

A load test with 200-token prompts and steady arrivals says your system handles 100 concurrent requests within SLO. In production, with 2,000-token prompts and bursty traffic, it saturates at 45. Why the gap?

The test measured an easier system than production. Longer prompts mean far more prefill compute (and more KV cache) per request, so the compute- or memory-bound knee arrives at much lower concurrency — the 200-token test never stressed the resource that 2,000-token traffic saturates. And bursty arrivals saturate earlier than steady ones, because bursts pile up faster than the server drains them, whereas the smooth test gave the system slack it won't have in production. The fix is to load-test with representative traffic — production-like prompt/output lengths and arrival burstiness — so the measured knee reflects the real workload, not a convenient one.

During a load test, at concurrency 60 throughput is still rising and p99 is stable; at 90 throughput is flat and p99 has tripled. Where is the knee, and what's your operating point?

The knee is around 60–80: throughput was still climbing at 60 (below the knee) and had plateaued by 90 while tail latency exploded (past the knee, into saturation), so the knee sits between them — you'd refine with a step around 70–80. Your operating point is just below the knee (roughly 60–70), where you get near-maximum throughput while p99 stays within SLO and there's headroom to absorb bursts. You would not run at 90: it delivers no more throughput than the knee and violates the latency SLO. Feed the knee value and the binding resource into capacity planning to size for peak load with headroom.

Related chapters

• Throughput and concurrency — the knee load testing locates
• Queueing and saturation — the runaway a load test drives into
• Representative test construction — the same realism lesson, for datasets
• Capacity planning and SLOs — turning the measured knee into a capacity plan