Part 8 — Production Observability

Request logging and traces

Observability foundations·Serving·6 min read

Observability is only as good as what you logged. This chapter is the foundation: capturing every request as a structured trace — a reconstructable record of what happened across every stage — so that when a novel failure appears, you can open the exact case and read it, the way failure analysis taught you to.

Observability is the ability to reconstruct and investigate an unanticipated failure — and you can only reconstruct what you recorded. So the foundation of the whole part is unglamorous: log every request as a structured trace. A trace is the full, reconstructable record of one request as it moved through the system, and it is the production equivalent of the trial-level trace you opened during failure analysis. Get this right and every later capability — quality signals, drift, incident investigation — has something to stand on. Skip it and you're debugging production blind.

What you will understand by the end

  • What a trace is and why it's the atom of observability.
  • What to capture per request so a failure is reconstructable.
  • Why traces must be structured and correlated, not scattered log lines.
  • Why you instrument for reconstruction before the incident, not during it.

A trace is a reconstructable record of one request

A trace captures everything needed to replay and understand a single request end to end: its identity, its inputs, what each stage produced, the final output, and metadata. It's the same stage-by-stage record that made offline failure analysis possible — a failure at any stage looks like "wrong answer," so you need each stage's input and output to localise it. In production, that record exists only if you emitted it.

Key idea

The trace is the atom of observability: one request, fully reconstructable. Everything else in this part — quality signals, drift, incident investigation — is analysis over traces. A system that logs only "request received / request completed" can tell you it failed but never why; a system that logs a full stage-by-stage trace lets you open the exact failing case and read it.

What to capture

A trace worth having records, at minimum:

  • A correlation id — a unique request id threaded through every stage and log line, so the pieces of one request can be reassembled (and tied to a user session, carefully — see privacy).
  • Inputs — the original request and relevant context.
  • Each stage's output — retrieval results, the assembled prompt, the raw model output, the parsed object, the validation result, any correction, the executed result. (The LLM-specific ones get their own chapters next.)
  • The outcome — success/failure, any error, and which system version served it.
  • Timing and metadata — per-stage latency, model id, tokens, timestamps.

The test of a good trace: could you reconstruct and re-run this request from it alone? If not, you'll be guessing when it fails.

Watch out

Unstructured, uncorrelated log lines are almost useless for LLM debugging. If a request's stages scatter across log statements with no shared id, you can't reassemble what happened to one request among thousands interleaved. Emit structured traces with a correlation id, not free-text prints — the difference is between "open the exact failing case" and "grep and pray."

Instrument before you need it

The cruel asymmetry of observability: you can only investigate what you already logged. When a novel failure appears, it's too late to add the instrumentation that would have explained it — the failing requests have already gone by unrecorded. So tracing is not something you bolt on during an incident; it's built in from the start, so that whatever goes wrong, the record already exists. This is also what makes the incidents-to-tests loop possible: an incident can only become a test case if you captured the failing input.

Observed evidence

This project's dashboard is trace-first by design: every one of the trials is an openable, structured record — the question, each system's raw and corrected intent, the expected answer, the outcome and split — which is exactly why failure analysis was possible at all. Turn that same trace-first habit onto live traffic and you have production observability; the offline trials show what a good trace lets you do. Open a structured trace →

Mental model

A trace is the reconstructable record of one request across every stage — the atom of observability and the production twin of the failure-analysis trial. Capture a correlation id, inputs, each stage's output, the outcome, version, and timing, structured so you could re-run the request. Instrument before the incident, because you can only investigate what you already logged.

Common mistakes

  • Logging only start/end. You learn that a request failed, never where or why — no reconstruction possible.
  • Unstructured, uncorrelated logs. Without a shared request id and structure, you can't reassemble one request from thousands.
  • Skipping intermediate stages. A failure at any stage looks like "wrong answer"; without per-stage records you can't localise it.
  • Adding tracing during an incident. The failing requests already passed unrecorded; instrument beforehand.

Practical guidance

  • Emit a structured trace per request with a correlation id threaded through every stage and log line.
  • Capture inputs, each stage's output, the outcome, the system version, and timing — enough to reconstruct and re-run the request.
  • Build tracing in from the start, sampling volume if needed, so the record exists before any failure you didn't predict.
  • Make traces easy to open and filter (by outcome, version, time) — friction here is friction on every future investigation.

Summary

  • A trace is the reconstructable, stage-by-stage record of one request — the atom of observability and the production twin of the trial trace.
  • Capture a correlation id, inputs, each stage's output, the outcome, version, and timing — the test is "could I re-run this request from the trace?"
  • Traces must be structured and correlated, not scattered log lines.
  • Instrument before the incident — you can only investigate what you already logged, and only captured failures can become test cases.

Knowledge check

A novel failure hits production. Your logs show "request 8471 received" and "request 8471 completed (error)" but nothing in between. What can you learn, and what does that teach about instrumentation?

Almost nothing useful: you know request 8471 failed, but not where or why — was it retrieval, the model, parsing, a tool call? Because a failure at any stage looks the same at the boundary, you can't localise it without each stage's input and output, which you didn't capture. The lesson is that observability requires instrumenting for reconstruction beforehand: you can only investigate what you already logged, and by the time this novel failure appeared it was too late to add the tracing that would have explained it. Structured, stage-by-stage traces with a correlation id must be built in before the incident, not after.

Why is a correlation id threaded through every stage essential in a high-traffic LLM system?

Because thousands of requests are interleaved, and one request's stages (retrieval, prompt, model, parse, validate, execute) are emitted as separate events. Without a shared correlation id you can't tell which retrieval result, prompt, and model output belonged to the same request, so you can't reassemble the trace for the specific failing case — you'd be grepping through a mixed stream. The correlation id is what lets you reconstruct one coherent request-journey out of the interleaved flood, which is the whole point of a trace.

Related chapters