Part 2 — Complete LLM Systems

Prompts, schemas, parsing and validation

The system, not the model·Core·7 min read

The model sits between an input side and an output side, and both are ordinary software you control. This chapter builds out those two sides as first-class system components — prompt assembly on the way in, schema-constrained output and a parse-then-validate stage on the way out — because that's where most reliability is won.

The model is only one component established that a score belongs to a whole configuration. This chapter zooms into the two components that bracket the model: the input side (how the prompt is assembled) and the output side (how raw text becomes a checked, structured object). Neither involves the model at all — they're ordinary software you write — and yet they're where a large share of a system's reliability lives. Treating them as first-class, versioned components, not glue code, is a defining habit of a serious LLM system.

What you will understand by the end

  • The prompt as an assembled artifact — system, context, input, instructions — you version.
  • Why the output schema is a design decision, not an afterthought.
  • The parse → validate stage as the gate between the model and everything downstream.
  • Why these non-model components carry so much of the system's reliability.

The input side: the prompt is assembled, not written

A production prompt is rarely a fixed string — it's assembled per request from parts (anatomy): a stable system prompt (role, rules, output format), task-specific context (retrieved documents, a schema, a glossary), the user input, and instructions/examples. Each part is a place a bug can enter — a missing rule, an overflowing context window, a buried instruction — so the assembly logic deserves the same care as any other component: tested, versioned, and inspectable.

Key idea

The prompt is a program you assemble, and its assembly is real software with real bugs. Because the stable parts (system prompt, schema) repeat across requests, they're also where a change ripples widest — a one-line system-prompt edit can move behaviour on every request. Treat prompt assembly as a versioned component, not throwaway string-building.

The output side: schema as a design decision

Most systems need the model to produce something a program can consume, so you define an output schema (structured output) — fields, types, allowed values — and often enforce it with constrained decoding so the shape is guaranteed. But the schema is a design choice with consequences: an under-specified schema lets "valid" outputs still be unusable (ambiguous fields, loose types), while a well-designed schema narrows what the model can get wrong. Design the schema to make the right answer easy to express and wrong answers hard to represent.

The gate: parse then validate

Between the model's raw text and everything downstream sits the stage that makes an LLM system safe to build on: parse the raw output into a structured object, then validate that object against the schema and any business rules. This is a hard gate — nothing malformed or illegal should pass it into execution.

   raw model text ─▶ PARSE ─▶ structured object ─▶ VALIDATE ─▶ (schema + rules ok?) ─▶ downstream
                       │                              │
                   reject malformed              reject illegal
                   (or repair/retry)             (missing field, bad value)
Watch out

Two opposite failures live here. A parser/validator that's too lenient lets malformed or illegal output through, and a valid-but-wrong value becomes a downstream incident. One that's too strict rejects good output over harmless variation, manufacturing failures. And remember the boundary this stage does not cover: it checks shape and legality, never meaning — a schema-valid object can still say the wrong thing.

Reliability lives in the non-model components

The through-line of Part 3 is that the components around the model carry most of the reliability, and this chapter is the clearest case. The model produces one thing — raw text — and whether that becomes a dependable system depends on prompt assembly getting the input right and the parse/validate gate keeping bad output out. Both are deterministic software you fully control, which is exactly why they're the highest-leverage place to invest: you can't make the model perfect, but you can make everything around it correct.

Observed evidence

This project's design is this chapter in practice: the model produces a QueryIntent, which is parsed and validated against the schema before a deterministic builder turns it into a query — and constrained decoding drove structural failures to zero (100% schema-valid across all six systems). The remaining failures were purely semantic, precisely because the parse/validate gate had fully handled the structural boundary. The non-model plumbing did its job so cleanly that only meaning was left to evaluate. See the schema and the validated intent →

Mental model

The model turns an assembled prompt into raw text; everything else is software you control. Assemble the prompt as a versioned artifact, design the output schema so wrong answers are hard to represent, and gate the model's output through a parse-then-validate stage that rejects malformed and illegal output. Shape and legality live here; meaning is checked elsewhere.

Common mistakes

  • Treating prompt assembly as throwaway glue. It's real software with real bugs, and its stable parts ripple across every request.
  • Under-specifying the schema. A loose schema lets "valid" output still be unusable; design it to make wrong answers hard to represent.
  • No hard parse/validate gate. Passing raw model output straight downstream is how valid-but-wrong becomes a production incident.
  • A gate too strict or too lenient. Too strict manufactures failures on harmless variation; too lenient lets bad output through.

Practical guidance

  • Version prompt assembly like code — test it, inspect assembled prompts in traces, and treat system-prompt edits as wide-blast-radius changes.
  • Design the output schema to narrow the model's room for error; enforce it with constrained decoding where structure must be guaranteed.
  • Make parse → validate a hard gate: reject or repair malformed output, enforce schema and business rules, and never let unvalidated output execute.
  • Remember the gate checks shape, not meaning — pair it with a separate semantic check.

Summary

  • The prompt is an assembled, versioned artifact (system + context + input + instructions), and its assembly is real, buggable software.
  • The output schema is a design decision — shape it so wrong answers are hard to represent.
  • The parse → validate stage is a hard gate for shape and legality between the model and downstream code.
  • These non-model components carry most of the reliability — this project's clean parse/validate gate is why only meaning was left to evaluate.

Knowledge check

Constrained decoding gives you 100% schema-valid output. Why do you still need a distinct validation stage, and what does it (and doesn't it) cover?

Constrained decoding guarantees the output is schema-shaped, but a real validation stage also enforces business rules beyond the schema (cross-field constraints, allowed combinations, referential checks) and is the explicit gate where you decide what may execute. What it covers is shape and legality; what it does not cover is meaning — a perfectly valid, schema-legal object can still select the wrong thing. So you keep the validate gate for structural safety and pair it with a separate semantic-correctness check for whether the answer is right.

Why is a one-line edit to the system prompt a higher-risk change than editing a single user-facing string?

Because the system prompt is a stable part assembled into every request, so a change to it has a blast radius across the whole workload — it can shift behaviour on all requests at once, including ones far from what you were trying to fix. A single user-facing string affects one spot. That wide blast radius is exactly why prompt assembly should be versioned and its changes run through the regression suite and gate, treated as system-wide changes rather than minor tweaks.

Related chapters