Part 6 — Inference Fundamentals

Orchestration and disaggregation

GPU hardware and scaling·Advanced·6 min read

Understand the top rung of the stack: an orchestration layer that serves a model across many nodes, the three scale-only techniques it is the home for, and — most importantly — the gate that tells you when a workload is too small to need it.

An inference engine serves a model on a node. The last rung, NVIDIA Dynamo, serves a model across many nodes. It sits on top of the engines — they are its backends — and it is the software home for the handful of techniques that only pay off at large scale. The most valuable thing to learn here is not how to turn it on, but how to recognize when a workload has not earned it.

What you will understand by the end

What an orchestration layer adds on top of an engine.
The three scale-only capabilities Dynamo provides.
How the SLA planner autoscales prefill and decode independently.
The gate — the judgment call for when to use an engine directly instead.

Where it sits

        ┌──────────────────────────────────────────────┐
        │        Orchestration (Dynamo)                 │  KV-aware router · SLA planner
        │        coordinates across nodes               │  · disaggregation manager
        └───────┬─────────────────┬─────────────────────┘
          ┌─────▼─────┐     ┌─────▼─────┐
          │  engine   │     │  engine   │   backends: vLLM / SGLang / TensorRT-LLM
          │ (prefill) │     │ (decode)  │
          └───────────┘     └───────────┘
             node 1            node 2

Key idea

An engine handles one node's worth of serving — batching and the KV cache on the GPUs it owns. Dynamo is a distributed control plane above the engines: it routes requests across replicas, splits prefill from decode, and spans a single model across nodes. It doesn't replace the engine; it coordinates many of them.

The three capabilities — all techniques you've met

Dynamo is the orchestration home for the techniques that only earn their keep at scale:

KV-aware routing — keep the KV cache between requests and route a request to the replica that already holds its prefix, so a shared prefix is reused instead of recomputed (prefix caching, across a fleet).
Disaggregation — run prefill and decode as separate pools so each scales on its own bottleneck, with the KV cache transferred between them.
Multi-node parallelism — run one logical replica across 2+ GPU nodes (typically for large mixture-of-experts models that don't fit on one node).

The SLA planner: asymmetric autoscaling

The piece worth dwelling on is that Dynamo can autoscale prefill and decode workers independently, against separate targets:

   TTFT target (prefill)                 tokens/sec target (decode)
        │                                        │
        ▼                                        ▼
   ┌──────────┐     KV transfer over        ┌──────────┐
   │ PREFILL  │ ── NVLink / InfiniBand ───► │  DECODE  │
   │ workers  │        (KV cache)           │ workers  │
   └────▲─────┘                             └────▲─────┘
        │ scale on TTFT                           │ scale on tokens/sec
        └──────────── SLA-based planner ──────────┘

Key idea

Prefill and decode have different bottlenecks — prefill is compute-bound and governs TTFT; decode is bandwidth-bound and governs tokens/sec — so their load ratios drift apart with traffic. Scaling them as one pool over- or under-provisions a phase. The planner sizes each pool on its own metric, moving the KV cache between them.

The gate: when not to use it

The more scale you have, the more tools become worth their complexity.

Dynamo only unlocks value at scale, and it adds real operational surface area — a KV-aware router, an SLA planner, a KV-transfer fabric, and separate worker pools, each a new thing that can mis-scale or partition.

What this does not suit

For a small or moderate deployment, Dynamo is overhead, not benefit. Unless you're operating with enough volume for disaggregation and KV-aware routing to matter, you use the inference engine directly behind your own control plane. Recognizing that a workload hasn't earned the orchestration layer is the actual skill.

The gate — the more of these that are true, the more Dynamo earns its place:

□ Model big enough to need multi-node / disaggregation? (foundation-scale, 100B+)
□ Traffic big enough that disaggregation pays for its complexity? (very high volume)
□ Prefill-heavy enough that splitting prefill/decode helps?
□ Prompts share enough structure that KV-aware routing earns its keep?
→ mostly NO → serve on the engine directly.

Observed evidence

This project's serving system is a concrete "below the gate" case: a single model on a single replica on one GPU. It correctly runs on the engine directly — no orchestration layer — and the serving write-up names multi-replica routing and cross-node autoscaling as the open frontier, i.e. the point at which a Dynamo-class layer would start to matter. The gate, applied to a real system. See the single-replica serving design →

Common mistakes

Adopting orchestration too early. A single-node model gains little and inherits a distributed control plane's failure modes.
Ignoring the routing tension. KV-aware routing (send to the replica holding the prefix) fights even load-balancing (that replica gets hot); watch hit-rate and per-replica load together.
Treating KV transfer as free. Moving the cache between pools consumes interconnect bandwidth; under-provision the fabric and the seam becomes the bottleneck.

Practical guidance

Walk the gate before designing around Dynamo; for most workloads, an engine behind a standard autoscaler is the right control plane.
If you do adopt it, run two SLO dashboards (prefill TTFT and decode tokens/sec) plus router hit-rate and KV-transfer throughput.
Expect churn — it's the newest layer of the stack; pin versions and confirm a feature exists before building on it.
The high-value exercise here is judgment, not a deploy: for any workload, decide engine-direct vs orchestrated and justify it with the gate.

Summary

Engines serve on a node; Dynamo orchestrates across nodes, and engines are its backends.
It's the home for three scale-only techniques: KV-aware routing, disaggregation, multi-node parallelism — with an SLA planner that scales prefill and decode independently.
It pays off only at foundation-model scale and high volume; below the gate, an engine used directly is the right answer — and this project's single-replica system is exactly that case.

Knowledge check

In one line each: what does an inference engine do versus Dynamo, and what's the backend relationship?

An inference engine serves a model on a node (continuous batching + techniques as flags). Dynamo is a distributed control plane that orchestrates a model across many nodes (KV-aware routing, disaggregation, multi-node parallelism). The engines are Dynamo's backends — it sits on top and coordinates them.

The SLA planner scales prefill and decode workers separately. Why separately, and on which metric each?

Because they have different bottlenecks: prefill is compute-bound and governs TTFT; decode is bandwidth-bound and governs tokens/sec. Their load ratios drift independently with traffic, so a single pool mis-provisions one phase. The planner scales the prefill pool on TTFT and the decode pool on tokens/sec, transferring the KV cache between them.

A single dense model on one GPU is getting slow under load. Is Dynamo the fix?

No — it's below the gate. One node, one model, no multi-node or disaggregation need. The fix is engine-level (batch and memory flags, prefix caching, or more replicas behind a standard load balancer), not a distributed orchestration layer, which would add overhead and failure modes without unlocking its scale-only capabilities.