Part 8 — Production Observability

Retrieval and tool-call traces

Tracing LLM systems·Serving·7 min read

Extended systems — RAG and agents — have extra stages that fail in their own ways, and they're invisible unless you trace them. This chapter is about capturing what the retriever fetched and what tools the model called, so a failure in a multi-stage system can be localised instead of blamed on the model.

The moment a system gains retrieval, tools, or agents, it gains stages that the basic prompt/response trace doesn't cover — and those stages have their own, common failure modes. A RAG answer can be wrong because the retriever fetched the wrong document; an agent can fail because it called the wrong tool with the wrong arguments. Neither is visible if you only logged the final prompt and response. This chapter is about tracing those extra stages so that, in an extended system, you can still answer the failure-analysis question: which stage broke?

What you will understand by the end

  • Why extended systems need retrieval and tool-call traces on top of prompt/response.
  • What to capture for retrieval (what was fetched, and why) and tool calls (which, with what arguments, returning what).
  • Why these traces are what localise a failure in a multi-stage system.
  • Why agents especially need step-level traces, not one record.

Retrieval traces: what was fetched, and why

In a RAG system, the retriever runs before the model and decides what context the model even sees — so retrieval quality is half the system. To debug it you must capture:

  • The query the retriever ran (which may differ from the user's raw input).
  • The documents/chunks returned, with their relevance scores and ranks.
  • Which of them made it into the prompt (after any reranking or truncation).

With this, a wrong RAG answer becomes diagnosable: if the retrieval trace shows the right document was never fetched, it's a retrieval failure (fix the index/query); if the right document was fetched and in the prompt but the model still answered wrong, it's a grounding/model failure (fix the prompt or model). Same wrong answer, opposite fixes — distinguishable only by the retrieval trace.

Key idea

A wrong answer in a RAG system is ambiguous until you see the retrieval trace: was the right context not fetched, or fetched and ignored? Those are different bugs with different fixes, and the retrieval trace — the fetched documents and their scores — is the only thing that tells them apart. Without it, every RAG failure gets blamed on "the model."

Tool-call traces: which tool, what arguments, what result

For tool-using systems and agents, the model emits tool calls, and each is a place to fail. Trace each call with:

  • The tool chosen and why it was available (so you can spot wrong-tool selection).
  • The arguments the model passed (wrong arguments are a common, invisible failure).
  • The result the tool returned (including tool errors, which the model then has to handle).

This turns "the agent gave a wrong answer" into "the model called search when it should have called calculate," or "it called the right tool with a malformed argument," or "the tool itself errored and the agent didn't recover" — three distinct, localisable failures.

Watch out

In an agent, the failure that surfaces is often several steps downstream of its cause: a bad tool result at step 2 quietly derails steps 3–8, and only step 8 looks wrong. Without a trace of every step's tool calls and results, you'll investigate the visible symptom (step 8) instead of the actual cause (step 2). Multi-step systems need multi-step traces, or root cause is a guess.

Agents need step-level traces

This is the production form of the Part 6 lesson that agents can't be scored with one number. An agent's decide → act → observe loop runs many steps, and a single end-to-end trace collapses them. You need the full step sequence — each step's reasoning, tool call, and observation — to see where a run went off the rails and why. The more steps and tools, the more a single record hides, and the more the trace has to preserve the whole trajectory.

Observed evidence

This project is deliberately single-step — question → validated intent → deterministic query, no retrieval or tools — which is exactly why its trace is simple: one raw output, one corrected output, one expected answer. That simplicity is the point of the design: every retrieval or tool stage you add is another thing to trace and another way to fail. The project shows the clean baseline; a RAG or agent version would need the extra retrieval and tool-call traces this chapter describes. The single-stage trace, with nothing extra to trace →

Mental model

Every extra stage — retrieval, each tool call, each agent step — is a new failure mode that's invisible unless traced. Capture what the retriever fetched (and its scores), which tools the model called with what arguments and results, and the full step sequence for agents. These traces are what localise a multi-stage failure to its actual stage instead of blaming the model — the production form of stage-level failure analysis.

Common mistakes

  • Only tracing prompt/response in a RAG system. You can't tell "wrong document fetched" from "right document ignored" — the two RAG failure modes.
  • Not logging tool arguments and results. Wrong-argument and tool-error failures become invisible and get blamed on the model.
  • One trace for a multi-step agent. It collapses the trajectory; you see the symptom step, not the cause step.
  • Blaming the model reflexively. In extended systems the cause is often retrieval or a tool, and only their traces show it.

Practical guidance

  • For RAG, trace the retriever query, returned documents with scores/ranks, and which made it into the prompt.
  • For tools, trace the tool chosen, arguments, and result (including tool errors) for every call.
  • For agents, capture the full step-level trace — reasoning, tool call, observation per step — not a single end-to-end record.
  • Use these traces to localise the failing stage before choosing a fix, the same discipline as offline root cause.

Summary

  • Extended systems add stages — retrieval, tool calls, agent steps — that fail in their own ways and are invisible unless traced.
  • Retrieval traces (fetched docs + scores) distinguish "not fetched" from "fetched and ignored"; tool-call traces (tool, args, result) distinguish wrong-tool, wrong-argument, and tool-error failures.
  • Agents need step-level traces — one record collapses the trajectory and hides the cause step.
  • These traces are the production form of stage-level failure analysis — they localise the fault instead of blaming the model.

Knowledge check

A RAG system returns a confidently wrong answer. Your prompt/response trace shows a reasonable prompt and a fluent answer. What extra trace do you need, and what two causes would it distinguish?

You need the retrieval trace — the query the retriever ran, the documents/chunks it returned with their relevance scores, and which made it into the prompt. It distinguishes two very different causes: (1) the right document was never fetched (a retrieval failure — fix the index or query), so the model answered from bad or missing context; versus (2) the right document was fetched and in the prompt but the model answered wrong anyway (a grounding/model failure — fix the prompt or model). The final answer looks identical in both, but the fixes are opposite, and only the retrieval trace tells them apart.

Why does an agent that fails at step 8 often require traces of steps 1–7 to diagnose?

Because in a multi-step agent the cause of a failure is frequently several steps upstream of the symptom: a wrong tool call or bad observation at, say, step 2 corrupts the agent's context and quietly derails everything after it, so only step 8 produces a visibly wrong result. If you only trace the final step, you'll investigate step 8 — which may look locally reasonable — and miss that step 2 is where it actually went wrong. The full step-level trace (each step's reasoning, tool call, and result) is what lets you walk back the trajectory to the real cause.

Related chapters