Part 1 — LLM Application Foundations

Application patterns: classification, extraction, RAG, tools, agents and NLQ-to-SQL

Prompts, schemas and application patterns·Core·5 min read

Recognise the handful of shapes almost every LLM application takes. Naming the pattern tells you the request shape, where the difficulty lives, and how you'll evaluate it — the bridge from 'what you're building' to the rest of the book.

Most LLM applications are one of a small number of patterns. Learning to name the pattern is worth a lot: it tells you the shape of the request, where the hard part lives, and — crucially for this book — how you will evaluate it and what will fail. This chapter is the map from "what you're building" to everything that follows.

What you will understand by the end

  • The common application patterns and what each one is.
  • Where the difficulty and the evaluation focus sit in each.
  • Why the running case study (natural-language-to-query) ties them together.

The patterns

Classification

Assign an input to one of a fixed set of labels (topic, sentiment, category, routing decision). Request shape: text in, a label out. Hard part: boundary cases and rare classes. Evaluated by: per-class precision/recall, not just aggregate accuracy.

Extraction

Pull structured fields out of unstructured text (names, dates, amounts, entities). Text in, a filled schema out. Hard part: messy inputs, missing or ambiguous fields. Evaluated by: field-level accuracy and whether it correctly abstains when a field isn't present.

RAG (retrieval-augmented generation)

Retrieve relevant documents, then generate an answer grounded in them. The model's knowledge is supplemented with fetched context. Hard part: retrieval quality and keeping the answer grounded (not inventing beyond the sources). Evaluated by: retrieval relevance and answer groundedness/faithfulness — two separate measurements.

Tools / function-calling

The model decides to call a function (search, calculator, database, API) with arguments, and uses the result. Language in, a structured call out, then an answer. Hard part: choosing the right tool and correct arguments. Evaluated by: tool-choice and argument correctness, and end-to-end task success.

Agents

Multi-step tool use with planning: the model loops — decide, act, observe, repeat — to complete a task. Hard part: compounding errors over steps, and knowing when to stop. Evaluated by: end-to-end task completion and step-level traces (a single final score hides where a multi-step run went wrong).

NLQ-to-SQL (natural-language-to-query)

Turn a plain-English question into a structured query over data. It combines the others: it's classification-like (which query pattern?), extraction-like (which metric, dimensions, filters?), and structured-output-heavy. Question in, a validated query intent out, then deterministic code builds the query. This is the pattern the whole book's case study uses.

Key idea

Naming the pattern is a shortcut to its evaluation and its failure modes. "It's a RAG system" immediately tells you to measure retrieval and groundedness; "it's an agent" tells you to keep step-level traces; "it's classification" tells you to check rare classes, not just the average. Pattern → what to measure.

The running case study

The evaluation study threaded through this book is an NLQ-to-SQL system: it reads a business question and produces a validated QueryIntent (metric, dimensions, filters, sort, limit), which deterministic code turns into a query. It's a good teaching vehicle because it contains the difficulties of several patterns at once — a boundary decision (ranking vs breakdown), field extraction (which metric/filter), and structured output — and every one of them shows up as something to evaluate.

Observed evidence

You can inspect this exact application as data: six complete systems on the same 88 questions, every trial openable to see what each produced. Open the trials where systems disagree →

Mental model

Most LLM apps are one of a few patterns — classification, extraction, RAG, tools, agents — and richer apps combine them. The pattern tells you the request shape, the hard part, and the evaluation. Recognise it first.

Common mistakes

  • Not naming the pattern. Without it, you evaluate by vibes and miss the pattern's known failure mode.
  • One aggregate score for a multi-part system. Agents and pipelines need step- or stage-level views, not a single number.
  • Treating RAG as "just prompting." Retrieval quality is half the system and needs its own metric.
  • Forcing the model to do deterministic work. In NLQ-to-SQL, the model produces intent; code builds the query — don't ask the model to emit SQL it can get subtly wrong.

Practical guidance

  • Classify your own app into a pattern (or a combination) before designing the eval.
  • Let the pattern pick the metric: per-class for classification, field-level for extraction, retrieval + groundedness for RAG, step traces for agents.
  • For combined patterns (like NLQ-to-SQL), evaluate each sub-difficulty separately.
  • Keep deterministic steps out of the model; evaluate the language-shaped step in isolation.

Summary

  • LLM applications cluster into a few patterns: classification, extraction, RAG, tools, agents — and combinations like NLQ-to-SQL.
  • The pattern determines the request shape, the hard part, and the evaluation.
  • The book's case study is an NLQ-to-SQL system precisely because it combines several patterns' difficulties — each one a thing to measure.
  • With Part 1 complete, you know what you're building; the rest of the book is how it runs, how to evaluate it, and how to keep it good.

Knowledge check

You're told an app is a RAG system. What two things must your evaluation measure, and why two?

Retrieval quality (did it fetch the right context?) and groundedness/faithfulness (is the answer supported by that context, not invented?). They're independent: retrieval can be perfect while the model still hallucinates beyond the sources, or the generator can be faithful to badly-retrieved context. One number hides which half failed.

Why is NLQ-to-SQL a good teaching pattern for evaluation?

Because it combines several patterns' difficulties in one task — a boundary/classification decision (which query shape), extraction (which metric, dimensions, filters), and structured output — so it exercises per-class metrics, field-level accuracy, and the valid-vs-correct distinction all at once, each of which becomes a concrete thing to measure.

Related chapters