Part 8 — Production Observability

Monitoring versus observability

Observability foundations·Core·6 min read

Offline evaluation ends where production begins. This part is about watching a live system, and it starts with a distinction that shapes everything: monitoring answers known questions with known metrics, while observability lets you ask new questions of a system that's failing in a way you never anticipated — which is most LLM failures.

The evaluation half of this book measured systems in the lab. But a deployed LLM system meets inputs no dataset contained, drifts over time, and fails in ways nobody wrote a test for. Watching it is its own discipline — production observability — and it opens with a distinction that decides how much you can learn when something goes wrong: monitoring tells you that a known thing broke; observability lets you figure out what an unknown thing is. LLM systems, whose failures are usually novel and semantic, need the second far more than most software does.

What you will understand by the end

  • The difference between monitoring (known questions) and observability (new questions).
  • Why LLM systems especially need observability, not just monitoring.
  • How this part relates to the offline failure analysis you already know.
  • The shape of the observability stack the rest of Part 8 builds.

Two different questions

  • Monitoring watches known signals for known failure modes: is the service up, is latency within SLO, is the error rate normal? You decide in advance what to measure and what threshold pages you. Monitoring is essential and answers "is the thing I worried about happening?"
  • Observability is the property that lets you ask questions you didn't anticipate. When a system fails in a new way, can you reconstruct what happened from what it emitted — traces, logs, the actual prompts and responses — well enough to understand a failure you never predicted? Observability answers "what on earth is going on?" for problems monitoring had no metric for.
Key idea

Monitoring answers pre-defined questions with pre-defined metrics; observability lets you answer new questions from rich, reconstructable output. Monitoring tells you something is wrong; observability tells you what. You need both — but the harder, more valuable one for LLM systems is observability, because their failures are the ones you didn't see coming.

Why LLM systems lean on observability

Traditional software fails in relatively enumerable ways — a null pointer, a timeout, a 500 — and monitoring catches most of them. LLM systems fail semantically and novelly: a valid-but-wrong answer, a new boundary confusion on an input type you never saw, a subtle degradation after a hosted model changed under you. None of these trips a "known metric" — the service is up, latency is fine, everything parses. The only way to catch and diagnose them is to have logged enough to reconstruct and investigate a failure you didn't anticipate. That's observability, and it's why "our dashboards are green" is never proof an LLM system is healthy.

Watch out

Green monitoring dashboards can sit on top of a badly broken LLM system: the service is up, latency is nominal, outputs are 100% schema-valid — and the answers are wrong. Monitoring only covers the failure modes you enumerated in advance, and LLM quality failures are exactly the ones you didn't. Don't mistake "no alerts firing" for "working"; you need observability to see the failures no metric names.

This is failure analysis, moved to production

You already know the core skill from Part 6: open the trial-level trace and read what actually happened stage by stage. Observability is that same skill, made possible in production — the difference is that offline you had the traces because you generated them, and in production you only have them if you instrumented the system to emit them. The rest of this part is about earning that ability: logging and traces, prompt/response capture, the LLM-specific metrics, quality signals without ground truth, drift, and doing it all without mishandling user data — then using it to investigate real incidents.

Mental model

Monitoring answers known questions with known metrics ("is the known thing broken?"); observability lets you answer new questions by reconstructing what happened ("what is this unfamiliar failure?"). LLM systems fail novelly and semantically, so green monitoring isn't proof of health — you need observability, which is failure analysis made possible in production by instrumenting the system to emit rich traces.

Common mistakes

  • Treating green monitoring as health. It only covers enumerated failures; LLM quality failures aren't among them.
  • Only monitoring, never instrumenting for observability. When a novel failure hits, you can't reconstruct it and you're guessing.
  • Assuming production failures look like traditional bugs. LLM failures are often semantic and novel — up, fast, valid, and wrong.
  • Waiting for an incident to add instrumentation. You can only investigate what you already logged; instrument before you need it.

Practical guidance

  • Run both: monitoring for known failure modes (uptime, latency, error rate) and observability for the unknown ones.
  • Instrument for reconstruction from day one — you can only diagnose a novel failure if the system already emitted enough to investigate it.
  • Never read "no alerts" as "healthy" for an LLM system; pair monitoring with quality signals and trace review.
  • Treat production observability as failure analysis you can run on live traffic — the same read-the-trace skill, earned by instrumentation.

Summary

  • Monitoring answers pre-defined questions with pre-defined metrics; observability lets you answer new questions by reconstructing what happened.
  • LLM systems fail novelly and semantically, so monitoring alone misses their most important failures — green dashboards can hide wrong answers.
  • Observability is failure analysis moved to production, possible only if you instrumented the system to emit rich traces.
  • The rest of Part 8 builds that capability — logging, prompt/response capture, metrics, quality signals, drift, privacy, and incident investigation.

Knowledge check

Your LLM service shows 100% uptime, latency within SLO, and 100% schema-valid outputs — yet a major customer says answers have been wrong for a week. How did monitoring miss it, and what would have caught it?

Monitoring only watches the failure modes you enumerated — uptime, latency, error/validity rates — and this failure is semantic: the answers are wrong while the service is up, fast, and valid, so no monitored metric moved. What would have caught it is observability plus quality signals: instrumented traces (the actual prompts and responses) to reconstruct and read the failing cases, and production quality signals (user edits, corrections, thumbs, sampled LLM-judge scores) that don't depend on a crash to fire. Green monitoring can't see wrong-but-valid; you need to be able to ask the new question "are the answers actually right?" — which is observability.

Why do LLM systems depend on observability more than typical CRUD web services?

Because typical web services fail in relatively enumerable ways (errors, timeouts, bad status codes) that monitoring can pre-define alerts for, whereas LLM systems fail semantically and novelly — valid-but-wrong outputs, new boundary confusions on unseen input types, subtle drift after a model change — failure modes nobody wrote a metric for. Catching and diagnosing those requires the ability to ask questions you didn't anticipate and reconstruct what happened, which is exactly observability. Monitoring handles the known; LLM systems generate an unusually large stream of the unknown.

Related chapters