CUDA and kernels

The hardware chapter told you what a GPU can do. This chapter is about the software floor that actually makes it do it — CUDA — and the one idea from it you will use constantly even though you will almost never write a line of it: kernel fusion, the code-level answer to decode being memory-bound.

What you will understand by the end

• What a kernel is, and why you select kernels rather than write them.
• Why a fast kernel is effectively locked to one GPU generation — a real risk.
• What kernel fusion does, and why it recovers decode throughput for free.
• Why torch.compile alone can't make LLM inference fast — the reason engines exist.

The serving stack, floor to ceiling

The whole serving stack stacks like this, and it's worth holding the whole ladder in mind as we climb it:

Orchestration   → coordinates a model across many nodes            (Dynamo)
Inference engine → serves a model on a node; techniques are flags  (the runtime)
Frameworks       → PyTorch + the formats weights live in           (PyTorch)
CUDA             → kernels that actually run on the GPU            ← you are here (the floor)
──────────────────────────────────────────────────────────────────
GPU hardware                                                       (the hardware)

CUDA is the proprietary platform that lets code run in parallel on NVIDIA GPUs. Its key unit is the kernel: a function that executes across thousands of GPU threads at once. Mentally, a kernel is just "a piece of code written for an NVIDIA GPU."

Kernels are hardware-locked

Here is the fact that makes CUDA a strategic concern and not just an implementation detail. Textbook attention is a few dozen lines of math. A production attention kernel like FlashAttention is the same math written as tens of thousands of lines of CUDA, hand-tuned to one GPU's exact memory hierarchy and tensor-core layout.

You select kernels, you don't write them

You will essentially never write a kernel. Decades of prebuilt libraries do it for you: cuBLAS and cuDNN provide the standard linear-algebra and neural-net primitives, and GEMM (general matrix-matrix multiply) — the operation every linear layer is — comes from them by default. Higher-performance building blocks (CUTLASS, FlashInfer) exist for the kernels that need hand-tuning.

Kernel selection — choosing the best available kernel for your model, GPU, and precision — is the real skill, and it's mostly automatic: engines ship pre-tuned kernels per architecture, and compilers pick kernels during their compile step. You occasionally select one explicitly with an engine flag (you'll see exactly this in Inference runtimes).

Kernel fusion: the decode-bandwidth fix, in code

Running two kernels back to back wastes a trip to memory: the first writes its result to VRAM, the second reads it straight back. Fusion collapses them into one kernel so the intermediate never touches VRAM.

UNFUSED:  read x → ×2 → WRITE tmp → READ tmp → ×3 → write out    (a wasted VRAM round-trip)
FUSED:    read x → ×6 → write out                                (same result, half the traffic)

There is no extra math in the fused version — the entire win is eliminating the write-then-read. Recall from Prefill and decode that decode is memory-bandwidth-bound: it spends its time moving bytes, not doing math.

Why this forces you up to an engine

Fusion happens two ways: compilers (torch.compile) auto-fuse straightforward, lightweight kernels; the sophisticated kernels (FlashAttention, custom GEMM) are hand-written and shipped as plugins.

Common mistakes

• Expecting torch.compile to deliver LLM speed. It can't fuse the plugin kernels that matter; you need an engine.
• Assuming a new GPU is automatically faster. Without tuned kernels it can be correct-but-slow — verify, don't assume.
• Trusting logs over benchmarks. If no native kernel exists for your precision on your card, the op silently falls back to a slow emulated path; only a benchmark reveals it.

Practical guidance

• Treat kernels as something you confirm and select, never write: check that your engine ships tuned kernels for your exact GPU and precision.
• When you hear "fusion," think bandwidth win — most valuable on the decode-heavy path.
• Pin official engine/driver container images so CUDA, drivers, and kernels arrive pre-wired and matched — sidestepping the day-zero-support trap.
• To feel this for yourself, write a fused-vs-unfused microbenchmark (multiply-by-2 then by-3 as two kernels, vs a single multiply-by-6) and time the wasted VRAM round-trip — the gap is pure memory traffic, not math.

Summary

• A kernel is GPU code; high-performance kernels are locked to a GPU generation, making day-zero support a real risk.
• You select kernels (mostly automatically), you don't write them.
• Kernel fusion removes wasted VRAM round-trips → free decode throughput; that's why FlashAttention exists.
• torch.compile can't fuse the plugin kernels the LLM hot path needs — which is precisely why you move up to an inference engine.

Knowledge check

A teammate says "let's just torch.compile our model and FlashAttention will get fused in automatically." What's wrong, and what's the fix?

torch.compile cannot fuse plugin kernels, and FlashAttention is a hand-written plugin — so compilation won't pull it in; it only auto-fuses simple, lightweight kernels. The fix is to use an inference engine (vLLM/SGLang/TensorRT-LLM) that already integrates FlashAttention and the other custom kernels.

Why is a high-performance kernel effectively locked to one GPU generation, and what should you check before promising a next-gen GPU will be faster?

It hard-codes hardware specifics (bandwidth, tensor-core count/layout), so it won't exploit — or may not even run on — a different architecture. Before promising a speedup on new silicon, confirm the engine ships tuned kernels for that architecture and your target precision; otherwise you get correctness without the performance.

Related evidence

• Decode dominates latency — the measured breakdown fusion targets →
• Forward: the engines that wire these kernels together →