Part 2 — Complete LLM Systems

Hosted versus self-hosted systems

Reliability and configuration·Serving·7 min read

Where the model runs is a system decision with wide consequences — control, cost, latency, privacy, and how much of the serving stack you own. This chapter frames the hosted-vs-self-hosted choice honestly: not 'which is better,' but which set of trade-offs fits your workload, scale, and constraints.

Every LLM system runs its model somewhere, and there are two broad answers: call a hosted model behind someone else's API, or self-host on your own hardware using the serving stack from Part 2. This is one of the most consequential architecture decisions you'll make, and it's often made on reflex ("just use the API" or "we must own it") rather than on the trade-offs that actually matter. This chapter lays those trade-offs out so the choice matches your workload instead of your instinct.

What you will understand by the end

  • What hosted and self-hosted each mean, and what you own in each.
  • The axes the decision really turns on: control, cost, latency, privacy, and operational burden.
  • Why the answer is workload- and scale-dependent, not universal.
  • Why it's a reversible, per-workload decision, not a one-time identity.

Two models of ownership

  • Hosted (API). You send requests to a provider's endpoint and pay per token. You own none of the serving stack — no GPUs, no engine, no scaling — and get frontier models, day-zero updates, and elastic capacity instantly. You give up control: the model can change under you, you can't tune the serving, and your data leaves your environment.
  • Self-hosted. You run the model yourself on your own (or rented) GPUs with an inference engine. You own the whole stack — hardware, engine, scaling, uptime — and with it full control: fixed model version, tunable serving, data that never leaves. You take on the operational burden that Part 2 described in full.
Key idea

The choice is a trade of control and cost-at-scale against operational burden and convenience. Hosted gives you frontier capability with zero ops and a per-token bill; self-hosting gives you control, data residency, and cheaper tokens at high volume — in exchange for owning the entire serving stack. Neither is "better"; they sit at opposite ends of an ownership spectrum.

The axes that decide it

  • Control. Hosted models can change or be deprecated under you (a real reproducibility and versioning problem); self-hosting pins the exact weights and lets you tune decoding, quantization, and batching.
  • Cost. Hosted is cheap to start (pay per token, no fixed cost) and can get expensive at high, steady volume; self-hosting has a large fixed cost (GPUs, ops) that amortises into cheaper tokens only above a break-even volume. The crossover is a real calculation, not a slogan.
  • Latency. Hosted adds network round-trips and shared-tenancy variance; self-hosting can hit tighter, more predictable tail latency and keep data local — if you engineer the serving well.
  • Privacy / data residency. Hosted sends your inputs to a third party (often unacceptable for regulated or sensitive data); self-hosting keeps everything in your environment.
  • Operational burden. Hosted has essentially none; self-hosting means you own GPU capacity, scaling, upgrades, and uptime — the entire Part 2 stack, forever.
Watch out

"Self-hosting is cheaper" is only true above a break-even volume, and the comparison must include the operational cost — engineers, on-call, GPU utilisation — not just the token math. At low or spiky volume, idle GPUs make self-hosting far more expensive per useful token than an API. Do the real break-even calculation for your traffic before claiming either is cheaper.

It's per-workload and reversible

This isn't a one-time identity ("we're a self-hosted shop"); it's a decision per workload, and it can change. A common, sensible pattern: prototype on a hosted API (zero ops, prove the task), then self-host if and when volume, cost, control, or privacy justify owning the stack. Different workloads in the same company can land differently — a sensitive high-volume path self-hosted, a spiky low-volume feature on an API. Decide it against each workload's constraints, and revisit as they change.

Observed evidence

This project spans both ends on the same task: the six systems include hosted frontier models (Claude, OpenAI) and self-hosted open models (Qwen 3B/7B on a single L4). That's the decision made concrete — the hosted models led on raw accuracy, while the self-hosted open models plus a correction policy were the cost/control play. Which to run is exactly the trade this chapter frames, measured head-to-head rather than argued. Compare hosted and self-hosted systems on one task →

Mental model

Hosted trades control and volume-cost for zero operations and instant frontier capability; self-hosting trades operational burden for control, data residency, and cheaper tokens above a break-even volume. It's not "which is better" but which trade fits this workload's scale, latency, privacy, and cost — decided per workload, and reversible. Prototype hosted, self-host when it pays.

Common mistakes

  • Deciding on reflex. "Just use the API" or "we must own it" skips the trade-offs that actually matter for the workload.
  • Claiming self-hosting is cheaper without the break-even. It's only cheaper above a volume, and only if you count operational cost and GPU utilisation.
  • Ignoring hosted model drift. A hosted model can change under you, breaking reproducibility; self-hosting pins the version.
  • Sending sensitive data to an API by default. Privacy/residency constraints can rule hosted out regardless of cost.

Practical guidance

  • Decide per workload against its real constraints — scale, latency SLO, privacy, cost, and how much control you need — not as a company-wide identity.
  • Do the actual break-even calculation (token cost vs fixed GPU + ops cost at your real, including-spiky, volume) before claiming either is cheaper.
  • Prototype hosted to prove the task with zero ops; self-host when volume, cost, control, or privacy justify owning the serving stack.
  • Treat the model source as part of the versioned configuration — a hosted model that can change under you is a reproducibility risk to manage.

Summary

  • Hosted (API, no ops, per-token, less control) vs self-hosted (own the stack, full control, cheaper at scale, heavy ops) are ends of an ownership spectrum.
  • The decision turns on control, cost, latency, privacy, and operational burden — and is workload- and scale-dependent, not universal.
  • Self-hosting is only cheaper above a break-even volume, operational cost included.
  • It's per-workload and reversible — prototype hosted, self-host when it pays; this project runs both on one task.

Knowledge check

A team says "we'll self-host to save money" for a new feature with low, spiky traffic. Why might that backfire?

Self-hosting has a large fixed cost (GPUs, plus the engineering and on-call to run the serving stack) that only amortises into cheaper tokens above a break-even volume. At low, spiky traffic the GPUs sit idle much of the time, so the cost per useful token can be far higher than a pay-per-token API — and that's before counting the operational burden. For low or bursty volume, a hosted API is usually cheaper and simpler; self-hosting pays off for high, steady volume (or when control/privacy require it). They should do the real break-even calculation for their actual traffic, including ops cost, before assuming self-hosting saves money.

Your system must produce reproducible results for an audit, and you're using a hosted model. What's the risk, and how does self-hosting address it?

The risk is model drift: a hosted provider can update or deprecate the model under you, so the exact system that produced an audited result may no longer exist — the same input can yield different output later, breaking reproducibility. Self-hosting pins the exact weights (and lets you fix decoding settings), so the configuration is frozen and reproducible for as long as you keep it. Whichever you choose, the model source belongs in the versioned configuration; with a hosted model you must at least track model/version identifiers and accept that reproducibility depends on the provider keeping that version available.

Related chapters