Part 6 — Inference Fundamentals
Prefill and decode
Understand what a language model is as a runtime object — a pile of weights read from memory on every step — and why that single fact splits inference into a compute-bound phase and a memory-bound phase that you optimize in completely different ways.
Before you can serve a model well, you have to see it the way the hardware does: not as an intelligence, but as a large pile of numbers that must be read out of memory every time it takes a step. Almost every performance fact in inference falls out of that one sentence.
What you will understand by the end
- What a "forward pass" costs, and why a 500-token answer is 500 of them.
- How to convert billions of parameters into gigabytes of memory (and why "it fits" is meaningless without a precision).
- The master split — prefill vs decode — and why each is bound by a different resource.
- Why decode speed depends on bandwidth ÷ model size, not on raw FLOPS, and why batching is nearly a free lunch until it isn't.
Core concept: a model is weights you re-read constantly
A model is a stack of layers of learned weights. Running an input through all of them once is a forward pass, and a forward pass is mostly matrix multiplications plus an attention step. The fact that governs everything:
Every forward pass must read the model's entire set of weights out of GPU memory. A 27-billion-parameter model in 16-bit precision is ~54 GB streamed per pass — and LLMs are autoregressive, generating one token per pass, so a 500-token answer streams those 54 GB five hundred times.
That is why memory, not cleverness, is the usual bottleneck.
Parameters ↔ memory: always ask "at what precision?"
A parameter count (in billions, "B") only becomes a memory footprint (in GB) once you pick bytes per parameter — a precision:
| Precision | bytes/param | GB ≈ |
|---|---|---|
| FP16 / BF16 | 2 | 2 × (params in B) |
| FP8 / INT8 | 1 | 1 × (params in B) |
| INT4 / FP4 | 0.5 | 0.5 × (params in B) |
The easy anchor: at INT8, gigabytes ≈ the billions-of-params number. So a 27B model is 54 GB at FP16, 27 GB at INT8, 13.5 GB at INT4.
"This 671B model fits in a 512 GB box" is only true at roughly INT4 (~336 GB). At FP16 it is 1.34 terabytes. If anyone says a model "fits" without stating the precision, the claim is undefined — ask "at what quantization?"
Mental model: prefill vs decode
Text generation has two phases with opposite appetites, and telling them apart is half of inference engineering.
| Prefill | Decode | |
|---|---|---|
| Does what | reads the whole prompt at once, builds the KV cache | emits output tokens, one forward pass each |
| Parallel? | yes — all prompt tokens together | no — token N+1 needs token N |
| Bound by | compute (the math units are busy) | memory bandwidth (waiting to reload weights) |
| Governs | time to first token (TTFT) | tokens per second |
Prefill is a parallel batch job that saturates the math units; decode is a serial loop that starves them, spending its time streaming weights from memory. They are bound by different resources, so they are sped up by completely different changes.
Why decode is memory-bound
A GPU can only bottleneck on two things: compute (math ops/sec, FLOPS) and memory bandwidth (bytes/sec from HBM into the cores). The ratio of the two is the card's break-even "appetite":
ops:byte = peak FLOPS ÷ memory bandwidth (H100 FP16 ≈ 989 TFLOPS ÷ 3.35 TB/s ≈ 295)
To keep the card balanced, your workload must do ~295 math ops for every byte it reads from memory. In decode with a single request, each weight is read and used about once — roughly 1 op per byte — so the math units sit idle waiting on memory. That gives the napkin ceiling for one stream:
single-stream decode speed ≈ memory bandwidth ÷ model size
27B FP16 on H100: 3,350 GB/s ÷ 54 GB ≈ 62 tokens/sec (for ONE request)
More FLOPS does not raise that number. A faster-compute card leaves single-stream decode speed unchanged — only more bandwidth, a smaller model, or fewer bytes per weight (quantization) moves it. This is the most common hardware-shopping mistake.
Batching: the (nearly) free lunch
The trick that rescues decode is batching — running many independent requests that share the same weights. One weight-load now serves the whole batch, so the math per weight-byte climbs with batch size (decode's arithmetic intensity ≈ batch size). Up to the break-even batch (~ops:byte), per-request speed stays roughly constant while aggregate throughput scales almost for free.
Continuous (in-flight) batching is the production form: requests join and leave the batch every token step, instead of waiting for a fixed group — which keeps the GPU full. (It descends from iteration-level scheduling; see Transformer and attention intuition.)
This project's serving experiment is a textbook case: throughput climbed from ~0.3 to ~7 req/s once concurrency actually reached the batch, and profiling showed decode is 99% of end-to-end latency while prefill is nearly free (served from cache). Exactly the prefill/decode asymmetry above, measured. Read the continuous-batching experiment →
The real ceiling is usually memory capacity
Batching is capped by a second memory fact: every concurrent request needs its own KV cache (the stored keys/values of prior tokens) alongside the weights. You almost always run out of capacity to hold KV caches before you run out of compute. PagedAttention packs the KV cache into non-contiguous pages so more requests fit — which is why it, not a faster chip, is often what raises real-world batch size.
Quantization: the triple win and its catch
Lowering weight precision (FP16 → INT8) moves half the bytes per pass, which improves three things at once, all for the same reason — decode is bandwidth-bound:
- latency: ~2× single-stream tokens/sec,
- throughput: frees VRAM for ~2× the KV-cache budget (bigger batch),
- cost: roughly half the $/token.
Quantization spends precision, which can hurt quality. INT8 is usually safe; INT4 starts to bite. The discipline is always the same: quantize, then measure quality on a real evaluation set before you ship it — which is exactly what the evaluation chapters of this book are about.
Common mistakes
- Quoting "it fits" without a precision. Off by up to 4× — the difference between a card and a cluster.
- Buying compute to fix decode. Decode is bandwidth-bound; FLOPS won't help.
- Sizing VRAM to the weights alone. Forgetting KV-cache capacity is how a model that loads still can't batch.
- Trusting a quantized model without re-evaluating. The three wins are real; the quality cost is invisible until you measure it.
Practical guidance
- Classify the workload as prefill-heavy (long inputs, short outputs) or decode-heavy (short inputs, long outputs) — that names the bottleneck.
- Estimate the model's footprint at your target precision, then leave KV-cache headroom on top.
- Reach for batching first (biggest throughput lever while memory-bound), then quantization (compounds), then a bigger card only if measurement demands it.
- After any of these, re-measure both speed and quality — never assume the theoretical win survived contact with your data.
Summary
- A model is weights re-read from memory every forward pass; a 500-token answer is 500 passes.
- Params → GB requires a precision; "fits" is undefined without one.
- Prefill is compute-bound (TTFT); decode is memory-bound (tokens/sec), with a
ceiling of
bandwidth ÷ model size. - Batching scales throughput nearly for free until the KV-cache capacity ceiling; quantization wins three ways but spends quality you must re-measure.
Knowledge check
A team upgrades to a GPU with 50% more FLOPS but the same memory bandwidth, hoping to speed up a chatbot's token stream. Will it work?
No. Single-stream decode is bound by bandwidth ÷ model size, not FLOPS.
More compute leaves per-token decode speed unchanged. They needed more bandwidth, a
smaller/quantized model, or larger batching — not more math.
Why does batching improve throughput "almost for free" up to a point?
Because decode is memory-bound: one weight-load serves the whole batch, so bytes moved stay ~fixed while useful math scales with batch size. Per-request speed holds roughly constant until the batch reaches the card's break-even (ops:byte) or runs out of KV-cache capacity.