GPU architecture and inference hardware

A GPU spec sheet lists a dozen numbers, and it is tempting to treat picking a card as a beauty contest between them. It is not. For any given workload, only one of three numbers actually binds — and once you know which, hardware selection collapses from guesswork into a one-line decision that the rest of your serving setup inherits.

What you will understand by the end

• The three spec-sheet numbers, and which phase of inference each one governs.
• How to place any workload on the "phase × resource" matrix and read off the bind.
• Why a bigger number within a generation does not buy you more compute.
• How to size VRAM for a model from memory.

Core concept: three numbers, three binds

Text generation has two phases with opposite appetites. Prefill reads the prompt in one big parallel pass; decode emits one token at a time, re-reading the model's weights on every step. Those two phases stress different parts of the card, and the KV cache they build stresses a third:

PREFILL  → COMPUTE   (FLOPS)      → governs TTFT (time to first token)
DECODE   → BANDWIDTH (TB/s)       → governs tokens/sec
BOTH     → VRAM capacity (GB)     → governs batch size / context length

A quick tour of each:

• Compute (FLOPS). The tensor cores do the matrix multiplies that are inference. Prefill saturates them; decode leaves them mostly idle until the batch is large. Two traps when comparing FLOPS: always use the dense figure (not the ~2× "sparse" number), and only compare at identical precision — FLOPS doubles every time precision halves (FP16 → FP8 → FP4), so an FP8-vs-FP16 comparison is rigged.
• Memory bandwidth (TB/s). Decode re-reads every weight for every token, so its ceiling is tokens/sec ≈ bandwidth ÷ model size. Prefill barely cares.
• VRAM capacity (GB). Holds the weights and the KV cache. Too little and the model will not load at all; more importantly, each concurrent request holds its own KV, so capacity is usually what caps your batch size.

Mental model: the phase × resource matrix

	Compute (FLOPS)	Bandwidth (TB/s)	VRAM capacity (GB)
Prefill	bottleneck — governs TTFT	slack	writes the KV cache
Decode	idle at low batch	bottleneck — governs tok/s	consumes the KV cache → caps batch

Read it by column: compute is lit by prefill only; bandwidth by decode only; capacity is touched by both (prefill produces KV, decode holds it). Whichever cell your traffic lights up reddest is the number you buy for.

The card line-up, in one rule

Within NVIDIA's line-up, two things move independently:

• The letter is the architecture generation and it is meaningful — Ada Lovelace (L4, L40), Hopper (H100, H200), Blackwell (B200, B300). A new letter is a real jump in compute.
• The number is size within a generation and it is nearly arbitrary — H100 → H200 is the same generation with more memory and bandwidth.

One more lever: NVLink, the fat GPU-to-GPU pipe (Hopper ~900 GB/s, Blackwell ~1.8 TB/s) that lets a node of 8 cards behave roughly like one big GPU — the thing that makes splitting a too-large model across cards viable. The cheap Ada cards (L4, L40) have no NVLink, which makes them a dead end for anything large.

Sizing rule of thumb: VRAM ≥ weights + at least 50% of the weight size as headroom for the KV cache (more for long context or high batch). A 54 GB model wants ~80 GB of card before you have run a single request.

Where this meets real measurement

Theory says which number should bind. A benchmark says which one does — and the two do not always agree, which is exactly why you measure.

That result also shows why the "measure, don't assume" habit matters: the same experiment found that raising the context-window setting was free on this card (huge KV headroom) and that turning off prefix caching was catastrophic (3–4× less throughput) — neither of which the spec sheet tells you.

Common mistakes

• Comparing FLOPS across precisions or dense-vs-sparse. The bigger number wins on paper and means nothing. Normalise first.
• Buying a within-generation upgrade for a compute problem. H100 → H200 will not speed up a prefill-bound job; you needed the letter to change.
• Sizing VRAM to the weights alone. Forgetting KV-cache headroom is how a model that "fits" OOMs the moment you batch it.
• Assuming the theoretical bind is the real bind. It often is — but only a benchmark settles it, and the exceptions are where the money is.

Practical guidance

• Classify the workload first: is it prefill-heavy (long inputs, short outputs) or decode-heavy (short inputs, long outputs)? That names the binding number.
• Normalise every spec you compare to dense FLOPS at one precision.
• Size VRAM as weights + KV headroom, and confirm the real batch you can hold.
• Then measure — the knee, the KV usage, the tail latency — before you scale the card up. The cheapest upgrade is often a config change, not a bigger GPU.

Summary

• Three numbers bind inference: FLOPS (prefill/TTFT), bandwidth (decode/tok-s), VRAM capacity (batch/context). Buy for the one your workload lights up.
• The letter buys compute; the number buys memory + bandwidth. NVLink buys the ability to go multi-GPU.
• The theoretical bind is a hypothesis; a benchmark is the answer — and on this project's L4 the measured knee was compute-bound, exactly as the matrix predicted.

Knowledge check

A workload has long document inputs and one-line outputs. Which number binds, and would you upgrade H100 → H200 or H100 → B200?

It is prefill-heavy, so compute (FLOPS) binds and governs TTFT. H100 → H200 is a within-generation bump that only adds bandwidth and VRAM, which are not your constraint. You need the letter to change: H100 → B200.

Two GPUs list similar FLOPS. What must you check before believing they are comparable?

That both figures are dense (not sparse) and quoted at the same precision. FLOPS roughly doubles each time precision halves, so an FP8-vs-FP16 or dense-vs-sparse comparison is not apples to apples.

Related evidence

• The serving-knobs experiment — the compute-bound knee on a real L4 →
• The systems compared in this project (and the hardware each runs on) →