Part 1 — LLM Application Foundations

Tokens, context and generation

From software to LLM engineering·Core·5 min read

Learn the three units every LLM application is actually measured in — tokens, the context window, and one-token-at-a-time generation — because cost, latency, limits, and even non-determinism all follow from them.

Before you can reason about an LLM application's cost, speed, or limits, you need the units the model actually works in. It does not read words or sentences; it reads and writes tokens, inside a finite context window, one token at a time. Almost every practical constraint in this book follows from those three facts.

What you will understand by the end

  • What a token is, and why cost and limits are counted in tokens, not words.
  • What the context window is, and why it's a budget you manage.
  • How generation works (prompt in, one token out at a time) — and why the same input can yield different outputs.

Tokens

Models don't operate on characters or words; text is first split into tokens — sub-word chunks. Common words may be one token; rarer words split into several. A useful rough anchor for English is ~4 characters per token, or ~0.75 words per token, but it varies by tokenizer and language.

Key idea

Tokens are the unit of everything that matters operationally: context limits are in tokens, API pricing is per token, and latency scales with tokens. "How long is this prompt?" is always really "how many tokens?"

Two practical consequences: counting tokens (not words) is how you predict cost and whether input fits; and non-English or code-heavy text can tokenize very differently, so never assume a word count.

The context window

Everything the model can "see" for one request — the system prompt, retrieved context, the conversation so far, the user's input, and the space reserved for its output — must fit inside a fixed context window measured in tokens.

Watch out

Input and output share the same budget: input tokens + output tokens ≤ context window. A long prompt leaves less room to generate; ask for a long answer with a long prompt and you can overflow the window (an error) or truncate the output (a broken response). Context is a budget you actively manage, not infinite memory.

This is why applications do things like retrieve only the most relevant context, summarise long histories, and cap output length — they are managing a finite token budget. Bigger windows exist, but a larger window is capacity planning, not a free pass: more tokens cost more and can be slower.

Generation: one token at a time

An LLM generates autoregressively — it produces the next token, appends it, and repeats, each token conditioned on everything before it. That splits a request into two phases you'll meet again in depth in Prefill and decode:

  • Reading the prompt (prefill): the whole input is processed to set up state.
  • Writing the answer (decode): tokens are emitted one at a time until the model stops or hits the output cap.
Key idea

Because output is produced token by token, a longer answer literally takes more steps — output length is a latency and cost lever. And because the next token is sampled from a probability distribution, the same prompt can produce different answers.

Why the same input gives different outputs

At each step the model has a probability distribution over possible next tokens. How it picks — the sampling — is controlled by settings like temperature:

  • Temperature 0 (greedy): take the most likely token every time → the most repeatable output. Use it when you want determinism (e.g. an evaluation, or structured extraction).
  • Higher temperature: sample more randomly → more variety and creativity, but less repeatable.

So non-determinism isn't a bug; it's a dial. When you need reproducible behaviour — as an evaluation does — you turn it down and pin the settings.

Mental model

The model reads and writes in tokens, inside a finite window, one token at a time. Cost, latency, context limits, and repeatability are all consequences of those three facts.

Common mistakes

  • Estimating length in words. Token counts drive cost and limits; word counts mislead, especially for code or non-English text.
  • Forgetting output shares the window. Sizing the prompt to the window leaves no room to answer.
  • Leaving temperature high when you need consistency. Evaluations and structured tasks want low temperature and pinned settings.
  • Treating a bigger context window as free. More tokens cost more and can be slower; size context to what's needed.

Practical guidance

  • Count tokens (with the model's tokenizer) when budgeting cost or checking fit.
  • Reserve output headroom: keep prompt + expected output comfortably under the window.
  • For anything you need to reproduce or evaluate, set temperature 0 and pin decoding settings.
  • Manage context deliberately — retrieve the most relevant, summarise long histories, cap output length.

Summary

  • Text becomes tokens; cost, limits, and latency are all counted in them.
  • The context window is a shared token budget for input and output.
  • Generation is autoregressive and sampled, so output length drives latency and the same input can vary — a dial you lower for reproducibility.

Knowledge check

A 3,700-token prompt goes to a model with a 4,096-token window, and you ask for up to 512 output tokens. What happens?

It overflows: 3,700 + 512 = 4,212 > 4,096. Input and output share the window, so you'd get an error (or a truncated answer). You'd need a shorter prompt, a smaller output cap, or a larger window. This exact arithmetic is a common source of "why did my response get cut off?".

Your extraction task must be reproducible for evaluation. What sampling setting do you use and why?

Temperature 0 (greedy) with pinned decoding settings. It selects the most likely token each step, giving the most repeatable output — necessary so that an evaluation measures the system, not sampling noise.

Related chapters