Part 6 — Inference Fundamentals

Batching and concurrency

Batching and latency·Serving·7 min read

Understand why serving many requests at once is not a nicety but the only way to make a GPU economical for decode — and how continuous batching turns a bandwidth-bound trickle into throughput, at a cost you pay in latency and memory.

A GPU running one request at a time during decode is a sports car in a traffic jam: almost all of its thousands of cores sit idle, waiting on memory. Batching — serving many requests together — is how you fill those idle cores, and it is the difference between a GPU that costs a fortune per token and one that is economical. This chapter explains why batching is mandatory for decode, how continuous batching works, and the latency and memory price you pay for the throughput.

What you will understand by the end

  • Why decode wastes a GPU at batch size 1, and why batching fixes it.
  • The difference between static and continuous batching, and why continuous won.
  • The three-way trade between throughput, latency and memory.
  • Why batching is a whole-system property, not a single flag.

Why one-at-a-time wastes the GPU

Recall the two phases from Prefill and decode: decode emits one token at a time, and each step must re-read the entire model's weights from HBM to produce that one token. Reading gigabytes of weights to compute a single token's worth of math leaves the tensor cores almost entirely idle — decode is memory-bandwidth-bound.

Here is the crucial asymmetry: the weights you streamed from HBM for one request can serve many requests in the same pass. If eight requests are all decoding their next token, you read each weight once and use it eight times.

  BATCH 1                          BATCH 8
  read weights ──► 1 token         read weights ──► 8 tokens
  (bandwidth spent, cores idle)    (same bandwidth, cores now busy)
  ~1× useful work per byte read    ~8× useful work per byte read
Key idea

Decode's cost is dominated by reading the weights, and that read is shared across everyone in the batch. So throughput rises with batch size almost for free until something else (compute or KV memory) becomes the new bottleneck. Batching is not an optimisation you add later — for decode economics it is the whole game.

Static versus continuous batching

The naive way to batch is static: gather N requests, run them together start to finish, return all N. The problem is that requests finish at different times — one wants 20 tokens, another wants 800. With static batching the whole batch runs until the longest member finishes, and every request that finished early leaves its slot idle. Worse, new arrivals wait for the entire batch to drain before they can start.

Continuous batching (also called in-flight or iteration-level batching) fixes this by batching at the granularity of a single decode step instead of a whole request:

  STATIC BATCHING                       CONTINUOUS BATCHING
  ┌───────────────────────────┐         ┌───────────────────────────┐
  │ R1 ████████ done, idle... │         │ R1 ████████ done → R5 ██   │  slot reused
  │ R2 ████████████████████   │         │ R2 ████████████████████    │  immediately
  │ R3 ████ done, idle....... │         │ R3 ████ done → R6 ██████    │
  │ R4 ██████████████ done... │         │ R4 ██████████ done → R7 █   │
  └───────────────────────────┘         └───────────────────────────┘
   batch fixed; finished slots waste    finished slot filled next step;
   GPU until the slowest finishes       new arrivals join without waiting

Every step, the scheduler assembles whatever requests are currently active, runs one decode step for all of them, evicts any that just finished, and admits waiting arrivals into the freed slots. No request waits for the batch to drain, and no slot idles while work is queued.

Key idea

Continuous batching is why modern inference engines get high GPU utilisation under real, ragged traffic. It is the default in every serious serving engine — and the reason "throughput under load" is a property of the scheduler, not just the model.

The trade: throughput up, latency and memory up too

Batching is not free — you buy throughput with two other budgets:

  • Latency. A bigger batch means each individual request shares the GPU with more others, so its per-token time can rise; and requests may wait briefly to be scheduled. You are trading per-request speed for aggregate throughput.
  • Memory. Every request in the batch carries its own KV cache. The batch size you can actually run is capped by how many KV caches fit in HBM — so batching and the KV budget are the same constraint viewed twice.
    batch size ↑
        ├── throughput ↑↑   (until compute- or memory-bound)
        ├── per-request latency ↑
        └── KV-cache memory ↑  → eventually the hard ceiling

The art of serving is finding the batch size that maximises throughput while keeping tail latency under your SLO — and that knee is workload-specific, which is why you load-test rather than guess.

Observed evidence

In this project's serving sweep, throughput climbed with concurrency until a distinct knee, after which more load only grew the queue and tail latency — no extra tokens per second. The knee was compute-bound on the L4 (the KV pool was nearly empty), so the ceiling was the GPU, not memory. The whole point of the experiment was to find that knee by measurement. Read the serving-knobs experiment →

Mental model

Decode reads the weights once and can serve the whole batch from that read, so throughput rises with batch size until compute or KV memory runs out. Continuous batching keeps the batch full step-by-step under ragged traffic — bought with per-request latency and HBM you must budget.

Common mistakes

  • Benchmarking at batch size 1. It measures the worst case for GPU economics and tells you almost nothing about production throughput.
  • Assuming a bigger batch is always better. Past the knee you only add queueing and tail latency — throughput flatlines while p99 climbs.
  • Ignoring the KV ceiling. The batch you want may not fit in HBM; batching is capped by KV-cache memory, not by a config number alone.
  • Confusing static and continuous batching. If finished requests idle their slots, you are leaving most of the GPU on the table under real traffic.

Practical guidance

  • Turn on continuous batching (it is the default in mainstream engines) and set a max batch / max-num-sequences to bound memory and tail latency.
  • Find the throughput knee with a load test at rising concurrency; serve just below it.
  • Watch both GPU utilisation and KV-pool utilisation to know whether you are compute- or memory-bound — that decides whether more batch or a bigger card helps.
  • Protect tail latency with a request timeout and admission control; a full queue past the knee helps no one.

Summary

  • Decode is bandwidth-bound, and the weight read is shared across the batch, so batching is how you convert idle cores into throughput — mandatory, not optional.
  • Continuous batching schedules per decode step, reusing finished slots and admitting arrivals immediately, which is why engines stay busy under ragged load.
  • The price is latency and KV memory; the best batch size is the throughput knee under your SLO, found by load testing.

Knowledge check

Your GPU shows low utilisation and low tokens/sec, but requests are arriving steadily one after another. What is likely wrong and what is the fix?

You are probably serving requests one at a time (batch size ~1), so each decode step reads all the weights to make a single token and the cores sit idle — bandwidth-bound with nothing to share the read across. The fix is batching (ideally continuous batching) so multiple requests decode together and reuse each weight read, raising throughput without more hardware.

Why does static batching waste the GPU when request lengths vary a lot?

Because the whole batch runs until the longest request finishes; every request that completed earlier leaves its slot idle for the remainder, and new arrivals must wait for the entire batch to drain. Continuous batching avoids both by reusing a finished request's slot on the very next step and admitting new work without waiting.

Related chapters