Part 1 — LLM Application Foundations

Prompts, schemas and structured outputs

Prompts, schemas and application patterns·Core·5 min read

Learn how you actually program an LLM — through the prompt and an output schema — and internalise the trap that runs through the rest of the book: constraining structure guarantees the shape of an answer, never its meaning.

You don't change an LLM's weights to make it do your task; you steer it with a prompt and shape its output with a schema. Together these are how you "program" the model. This chapter covers both — and the single most important caveat in applied LLM work: getting valid, well-structured output is not the same as getting a correct one.

What you will understand by the end

  • What goes into a prompt, and why the prompt is an interface you design.
  • What structured output is and why applications depend on it.
  • How constrained decoding guarantees valid structure.
  • Why valid JSON is not semantic correctness — the trap that motivates evaluation.

The prompt is the program

A prompt is usually assembled from a few parts:

  • System prompt — the model's role, rules, and output format ("You convert questions into intent JSON. Only use these fields…"). Stable across requests.
  • Context — task-specific facts: retrieved documents, a schema, a glossary, the conversation so far.
  • User input — the actual request.
  • Instructions / examples — how to do the task, sometimes with a few worked examples.
Key idea

The prompt is an interface you design, not a setting you tweak once. Clear instructions, explicit constraints, and a few good examples do more than a bigger model. And because a stable system prompt repeats across requests, it's also where later performance tricks (like caching a shared prefix) pay off.

Good prompting is mostly clarity: state the task plainly, enumerate the allowed outputs, give the format, and show an example or two. Vague prompts produce vague, unevaluable behaviour.

Structured output

Most applications need output a program can use, not a paragraph a human reads. So you ask the model for structured output — typically JSON matching a schema you define: fields, types, and allowed values.

Structured output is what lets the model's answer flow into the rest of the pipeline (the parse and validate stages from Anatomy): a QueryIntent object, an extracted record, a classification label, a tool call.

Constrained decoding guarantees the shape

You can go further than asking for JSON: constrained decoding (also called structured or guided decoding) restricts the model so it can only emit tokens that keep the output valid against the schema. Done this way, schema-valid output is guaranteed — no more "it mostly returns JSON but sometimes adds a sentence."

The trap that motivates the whole book

Here is the caveat to carry everywhere:

Watch out

Constrained decoding guarantees the shape of the answer, never its meaning. The model can emit perfectly valid JSON — right fields, right types — that says the wrong thing. Valid JSON is not semantic correctness.

Observed evidence

In the evaluation study used throughout this book, every system produced 100% schema-compliant JSON, yet their semantic accuracy ranged from about 45% to 96%. Structure was solved; meaning was not. That gap is the reason evaluation exists. See the systems compared →

So structured output solves the syntax boundary — the answer parses and integrates — but it does nothing for the meaning boundary. Separating those two is the foundation of everything in Parts 4–6, and it's stated as a system property in The model is only one component.

Mental model

You program the model through the prompt (what to do) and the schema (what shape to return). Structure buys you a parseable answer; correctness you still have to measure.

Common mistakes

  • Vague prompts. No enumerated outputs or format → behaviour you can't predict or evaluate.
  • Asking for JSON without enforcing it. Prompt-only requests drift; use constrained decoding when you need guaranteed structure.
  • Treating schema-valid as correct. The single most common applied-LLM error — valid output can be semantically wrong.
  • Under-specifying the schema. If the shape allows ambiguity, "valid" answers still can't be used reliably.

Practical guidance

  • Write the system prompt like an API contract: role, rules, enumerated outputs, format, one or two examples.
  • Use constrained decoding for anything a program consumes, so structure is guaranteed.
  • Then evaluate meaning separately — never let a schema check stand in for a correctness check.
  • Keep the shared parts of the prompt stable; it's both clearer and cheaper to serve.

Summary

  • The prompt (system + context + input + instructions) is how you program the model; design it like an interface.
  • Structured output + constrained decoding guarantee valid, usable shape.
  • Valid ≠ correct: structure is the syntax boundary; meaning is a separate boundary you must measure — the premise of the evaluation half of this book.

Knowledge check

Constrained decoding gives you 100% valid JSON. What have you proven about correctness?

Nothing. You've guaranteed the output's shape (fields, types), not its meaning. A model can emit valid JSON that selects the wrong metric or the wrong filter. Correctness is a separate property measured by evaluation — in the case study, 100% valid JSON coexisted with 45–96% semantic accuracy.

Why write the system prompt like an API contract?

Because clarity is the biggest lever: enumerating allowed outputs, the format, the rules, and a worked example makes behaviour predictable and evaluable, and a stable shared prompt is also cheaper to serve (its prefix can be reused). Vague prompts produce behaviour you can neither predict nor measure.

Related chapters