The model as parameters to be learned

You already know the Transformer's shape from the inference side: embeddings in, attention and MLP blocks in the middle, an LM head out. This chapter changes only your point of view. Instead of "weights I load and run," see the same structure as a pile of numbers that start out random and are gradually fitted by training. "Learning" is nothing more mystical than nudging those numbers until next-token prediction gets good.

What you will understand by the end

• What it means that a model "starts random" and where its parameters physically live.
• The parameter groups: token embeddings, positional embeddings, the blocks, the LM head.
• How the parameter count adds up, and the three knobs that scale it.
• The difference between parameters (learned) and hyperparameters (chosen).

Everything begins as random numbers

Before training, the model knows nothing: every weight matrix is filled with small random values. The forward pass still runs — tokens go in, logits come out — but the logits are noise, so the predictions are gibberish. Training is the process of repeatedly measuring how wrong those predictions are and adjusting the numbers to make them less wrong. When people say a model "learned" something, they mean these specific numbers settled into values that make the next token predictable.

Where the parameters live

Walk the same path a token takes, and note the trainable matrices it passes through:

• Token embeddings. A big vocab × d_model lookup table: one learned vector per token in the vocabulary. This turns a token ID into a vector and is often the single largest block of parameters. (The intuition — similar tokens end up with nearby vectors — is learned here, not hand-built.)
• Positional embeddings. A context × d_model table (or a rotary scheme like RoPE) that encodes where a token sits, since attention itself is order-blind. Added to the token embedding.
• The Transformer blocks. Each block holds the attention projection matrices (Q, K, V and an output projection — the subject of the next chapter) and the MLP's two big matrices (the expand-then-contract feed-forward), plus small normalisation parameters. Stacked N times, the blocks are usually the bulk of the count.
• The LM head. A final d_model × vocab projection that turns the last hidden vector into a logit per vocabulary token. (Often it shares weights with the token-embedding table.)

  token id ──▶ [token embed]  +  [pos embed]
                     │
                     ▼
              [ block 1 ] ─ attention (Q,K,V,O) + MLP + norms
              [ block 2 ]
                 ...            ← N stacked blocks = most of the parameters
              [ block N ]
                     │
                     ▼
               [ LM head ] ──▶ one logit per vocabulary token
   every box above is a matrix of numbers that starts random and gets trained

How "7B" adds up — the three knobs

The parameter count is not mysterious; it falls out of a few dimensions. Three knobs dominate it:

• Vocabulary size (V) — scales the token-embedding and LM-head tables (V × d_model each).
• Model width (d_model) — the vector size; it scales almost everything, and the MLP and attention matrices grow roughly with its square.
• Depth (N layers) — how many blocks you stack; the block cost multiplies by N.

Turn those up and the count climbs from "tens of millions" (a small model you can train on one GPU) to "billions" (a frontier model). Nothing else changed — same architecture, bigger dimensions, more numbers to fit.

Parameters versus hyperparameters

Two words that sound alike and mean opposite things:

• Parameters are the numbers the model learns — the embedding tables, attention and MLP matrices, the LM head. You never set these by hand; training sets them.
• Hyperparameters are the choices you make that shape training and the model: d_model, number of layers, number of heads, context length, and the training settings (learning rate, batch size). They are fixed before or around training, not learned.

Mental model

The architecture is a fixed skeleton of weight matrices; the parameters are the numbers in them, and they start random. Training only changes those numbers. The count is just vocabulary × width × depth — the same "billions" you sized VRAM against, seen from the inside.

Common mistakes

• Treating "learning" as adding knowledge or code. It only changes the values of a fixed set of numbers.
• Forgetting the embedding table's size. For small models the token embeddings can be the largest single block — vocabulary matters.
• Confusing parameters with hyperparameters. One is learned, the other is chosen; mixing them up muddles every training conversation.
• Assuming bigger architecture = better. More parameters only help if the data and training can fit them; capacity without good data overfits or wastes compute.

Practical guidance

• When you read "7B parameters," picture the matrices: embeddings + N blocks + head — and remember that is the exact number that had to fit in HBM at inference.
• Scale a model with intention: width buys per-token capacity (and grows cost quadratically), depth buys composition, vocabulary trades table size for shorter sequences.
• Decide hyperparameters deliberately and record them — they are the reproducible part of a model; the learned parameters are the output.
• Match parameter count to your data: a model far larger than its corpus can support will not learn more, it will just memorise.

Summary

• A model is its parameters — weight matrices that begin random and are fitted by training; the architecture is only the container.
• They live in token embeddings, positional embeddings, the stacked blocks, and the LM head, with blocks (and often embeddings) dominating the count.
• The count is set by vocabulary × width × depth — the very "billions" you sized VRAM against.
• Parameters are learned; hyperparameters are chosen — good training is largely choosing hyperparameters that let training find good parameters.

Knowledge check

A colleague says training "teaches the model new rules and adds capabilities." Reframe that in terms of parameters.

Training does not add rules or code — the architecture is fixed. It only changes the values of the model's existing weight matrices, starting from random. Any "capability" is those numbers settling into values that make next-token prediction accurate on the training distribution. Same container, different numbers.

You double d_model (the model width) and leave depth and vocabulary the same. Roughly what happens to the parameter count, and why?

It rises more than 2× — closer to 4× for the bulk of the model — because the attention and MLP matrices scale with the square of the width (they are roughly d_model × d_model shaped), while the embedding/head tables scale linearly with it. Width is the knob that grows parameters (and compute) fastest.

Related chapters

• Transformer and attention intuition — the same architecture, from the inference side
• Attention as learned Q, K and V — the block's attention matrices as trainable parameters
• Inside an NVIDIA GPU — why "how many parameters" is also a memory question
• Quantization — representing those same learned numbers in fewer bits