Part 6 — Inference Fundamentals

Model formats and frameworks

Runtimes and kernels·Advanced·6 min read

Understand the rung between raw CUDA and the engines: what PyTorch does and where its compilation stops, how weights are packaged (safetensors vs ONNX) and why that choice is both a security and a cold-start decision, and why the code on a model card is for learning, not for serving.

Between the CUDA floor and the engines sits the layer most people actually touch: PyTorch and the file formats your weights live in. Getting this rung right decides two things that bite in production — whether loading a model is a security risk, and how fast a new replica can start.

What you will understand by the end

The three paths from a trained model to a served one, and which one LLMs take.
Why safetensors beats a pickle/.bin — one security reason, one ops reason.
Why model-card sample code is a reference implementation, not a server.
Why cold-start time is dominated by moving weights, and how format affects it.

The fork: three paths from model to production

A model trained in PyTorch can be served three ways:

(a) Compiled PyTorch      (b) Export to an intermediate       (c) Straight to an
    (torch.compile)           representation                       inference engine
    max CONTROL               ONNX → ONNX Runtime / TensorRT       max CONVENIENCE
    rare/custom archs         PORTABLE; best for IMAGE/VIDEO       how most LLMs ship
                              but export breaks on novel ops

Key idea

For text LLMs, path (c) wins: ship the weights as safetensors and hand them to an engine. The ONNX/TensorRT middle path survives mainly for image/video generation models. Path (a) is for research and rare architectures. Knowing which path a workload is on is half of choosing its serving stack.

PyTorch — and where its compilation stops

PyTorch is the standard framework for both training and inference. It won largely on autograd (automatic gradients) — a training superpower irrelevant to serving but the reason it dominates. For inference, its bridge to speed is torch.compile, which does automatic kernel selection and fusion.

But we saw its ceiling in CUDA and kernels: torch.compile cannot fuse plugin kernels, and the LLM hot path is exactly those plugins. So pure-PyTorch compilation is good for rare architectures and long chains of lightweight ops — not the LLM fast path. That gap is what the engines fill.

Model formats: safetensors vs ONNX

	safetensors	ONNX
Holds	weights only	weights + execution graph
On load	no code execution; memory-mapped	bundles architecture with weights
Use when	the architecture already lives in code (the LLM norm)	you want to ship the graph too

Two facts make this a real decision, not a detail:

Watch out — supply chain

Generic .bin/pickle formats can execute arbitrary code during deserialization — loading untrusted weights is like running an unknown binary. safetensors holds only tensor data, with no code path that runs on open. Treat non-safetensors weights from an untrusted source as untrusted code.

Key idea

safetensors is memory-mapped: the loader reads tensors lazily instead of allocating the whole footprint up front. That directly shortens cold start — the time a new replica spends loading weights before it can serve — which is the dominant term when an autoscaler brings capacity online.

Observed evidence

This project's serving replicas scale to zero, and the first request after idle pays a roughly two-minute cold start — almost entirely weight-load and engine warm-up, not container start. That's why format and load-time levers matter operationally, not just aesthetically. See where cold start shapes the serving design →

Because load time scales with bytes, quantization pays a second dividend here: a model at 1 byte/param loads in roughly half the time of the same model at 2 bytes/param — faster scale-up on top of the runtime wins from Prefill and decode.

ONNX Runtime and TensorRT — the portable middle

ONNX Runtime (open, many hardware targets) and TensorRT (NVIDIA, strongest out-of-the-box for image/video) are high-performance runtimes you export a model into. Export is like compilation, but these standards don't support every PyTorch operation — a single novel op (some modern attention variants are hard to export) can block the whole path.

Watch out

Don't confuse TensorRT (this rung — an image/video runtime that exports an "engine" from ONNX) with TensorRT-LLM (the inference engine in Inference runtimes, whose current version dropped the TensorRT dependency entirely). Same brand, different layers.

`transformers` / `diffusers` are for learning, not serving

The Hugging Face transformers (LLMs) and diffusers (image/video) libraries are built on PyTorch but are explicitly reference implementations, not production servers. They're perfect for nailing a model's exact input/output spec and a quick local correctness check — and their config.json metadata and download utilities are genuinely useful.

Watch out

Every model card's sample code is transformers/diffusers, so it's tempting to ship it. But it has no continuous batching, no PagedAttention, no FlashAttention wiring — throughput is a fraction of an engine's. Spotting "they prototyped in transformers" and moving them to an engine is a routine, high-value correction.

Common mistakes

Shipping the model-card notebook. It defines the I/O spec; it is not a server.
Loading pickle/.bin from untrusted sources. That's arbitrary code execution on load — prefer safetensors.
Assuming a novel model exports cleanly. Validate ONNX/TensorRT export early; one unsupported op can block the plan.
Ignoring cold start. For bursty or scale-to-zero traffic, weight-load time is a first-class design constraint.

Practical guidance

For LLMs, take path (c): safetensors → engine. Reserve ONNX/TensorRT for image/video generation, and compiled PyTorch for rare architectures.
Prefer safetensors and treat other formats from untrusted sources as suspect.
Budget cold start explicitly — memory-map weights, cache them on fast local storage, keep warm capacity, and remember quantization halves the load.
Use model-card code only as the spec and sanity check, never as the benchmark target.

Summary

Three paths to production; LLMs take safetensors → engine.
safetensors is safer (no code execution) and faster to load (memory-mapped) than pickle — a security and a cold-start win at once.
transformers/diffusers teach the spec; they don't serve it.
Cold start is a weight-load problem, and format + precision are its levers.

Knowledge check

Give one security reason and one operations reason to prefer safetensors over a .bin/pickle file.

Security: pickle/.bin can execute arbitrary code during deserialization, so loading untrusted weights runs untrusted code; safetensors holds only tensor data. Operations: safetensors is memory-mapped, so weights load without allocating the full footprint up front — shortening cold start when a new replica comes online.

An engineer demos a working Hugging Face transformers notebook and wants to put it straight behind autoscaling. Why is that the wrong serving stack?

transformers is a reference implementation, not a server: it has no continuous batching, no PagedAttention, no FlashAttention wiring, so the GPU sits idle between requests and throughput is a fraction of an engine's. Keep the notebook as the I/O-spec reference and correctness check, and serve the model on an inference engine.