Agent Trajectory Observability: Judge the Path, Not Just the Answer
Two agents answer the same user query. Both return the identical string—correct, well-formatted, cited. An answer-level eval gives them both a perfect score, identical down to the decimal.
One agent made three redundant retrieval calls (same tool, same query, same corpus) before stumbling on the right source. The other called exactly the right tool once and answered. The answer-level eval cannot tell the difference. It never could.
The keys are in the trajectory.
I built a trajectory observability lane for my agents in three small pieces: the JSONL traces every workflow already emits but nobody reads, a judge that scores the tool-call sequence instead of the answer, and a Langfuse uploader written against the raw REST ingestion API—no SDK. Publication volumes indicate this is the moment: agent-observability research jumped sharply into 2026 (the phrase barely existed before), and the first dedicated fault-detection benchmark for agent observability was published this week.
This post is the full walkthrough: what trajectory observability is, why answer-level evals miss half the story, the three-module build, and how the research on partial observability validates the approach.
What Is Agent Trajectory Observability?
Agent trajectory observability is the practice of capturing, tracing, and analyzing the complete sequence of steps an AI agent takes to complete a task—including LLM calls, tool invocations, intermediate outputs, and error states. It answers how an agent reached its answer, not just what it answered.
In my stack, a trajectory is a sequence of JSONL events spanning the agent’s lifetime. Each event records a timestamp, run ID, workflow name, event type (e.g., tool_call), payload (tool, query, corpus), and elapsed milliseconds. The trace starts at the /research/agent endpoint and ends when the agent returns its final answer. Between those bookends, every tool call and retrieval is recorded.
The key insight: trajectory observability reveals the decision path, which answer-level evals intentionally discard. The research community has only recently formalized this distinction.
Why Agent Trajectory Observability Matters for AI Agent Reliability
Just days ago, “AgentTelemetry: A Fault Detection Benchmark and Toolkit for LLM Agent Observability” (Balusu, 2026) appeared in the Proceedings of the 3rd ACM International Conference on AI-Powered Software (AIware '26), pp. 380–387 DOI 10.1145/3805760.3814931. It defines a taxonomy of 14 fault types over 9 agent-specific span kinds and provides the first dedicated benchmark for detecting faults from agent telemetry. The benchmark is days old—details may shift in camera-ready—but the signal is clear: the research community is formally defining the problem I stumbled into with a JSONL file.
The volume trend backs this up. A strict OpenAlex title/abstract query for “agent trajectory” + observability returns roughly 1 publication in 2024, 0 in 2025, and ~39 in the first half of 2026 (counts are point-in-time and will drift as OpenAlex re-indexes—grouped by publication_year, 2026 sat at 41 when I last ran it). A looser OR-search runs into the thousands per year. The direction is robust: near-nonexistent before 2026, then the steepest relative growth of all eleven capabilities I have tracked. The exact multiplier is query-sensitive (some results are robotics and RL trajectory work that happen to match keywords), but the arc is unambiguous.
The idea that a path under partial observability is worth analyzing separately from its endpoint is not new—it is inherited from decades of reinforcement-learning and robotics work. Foerster et al.’s Counterfactual Multi-Agent Policy Gradients (COMA) (2018) uses a counterfactual baseline to assign credit when agents cannot see each other’s full trajectories, and Rudenko et al.’s human-motion trajectory-prediction survey (2020) treats missing and noisy observations of a path as the default case, not the exception. Agent trajectory observability is that same instinct, pointed at an LLM’s tool-call sequence.
One nearby example from my own frontier report is worth an honest caveat. It cites Lazem & Teahan’s Dynamic Bidirectional Pattern Memory: A Production-Scale Empirical Characterisation of Inference-Time Gating in Clinical NLP (July 2026)—but that paper is a clinical-NLP study of inference-time memory-gating (flagging suspect extractions in a production pipeline), not an agent-trajectory paper. It is an analogous case of decision traceability through gating, and I cite it as exactly that, not as evidence for trajectory evaluation.
Key Components of Agent Trajectory Observability
My implementation consists of three modules, each small enough to read in one sitting:
1. Trace capture. The workflow runtime writes one JSONL line per streamed event into .agent-traces/. The event tracer (rag_service/workflow_trace.py, JsonlEventTracer) is env-gated, zero cost when off. The observability gap was not missing instrumentation—the JSONL files were write-only.
2. The judge. rag_service/eval/agent_eval.py::judge_trajectory takes the step list and returns a machine-readable verdict: {score, redundantCalls, abstainedAppropriately, toolSequence, feedback}. Two checks run deterministically with zero credentials: redundant calls (same tool+query+corpus+filters twice) and appropriate abstention (all retrievals empty, and the agent says “insufficient grounding” instead of hallucinating). The judge scores abstention as a win—the opposite of answer-level evals, which punish it. An optional LLM leg refines the 0–1 score but never overrides the deterministic gates.
3. The drain. roadmap-kg/kg/trace_uploader.py batches trace lines into Langfuse’s public REST ingestion endpoint (/api/public/ingestion) using stdlib urllib. No SDK: my TypeScript side already talks to Langfuse via bare REST (lib/langfuse.ts), and both no-op cleanly when keys are unset. A per-directory ledger tracks uploaded lines, making reruns idempotent and partial uploads resumable.
How to Implement Trajectory Observability in Agent Systems
Here is the decision framework grounded in the evidence:
When to use it. Your agent has ≥2 tool calls in its typical execution path, or your agent spawns sub-agents, or you have observed failures that cannot be reproduced with the same input (non-deterministic tool selection).
What to check first. Deterministic checks for redundant calls and appropriate abstention. Both are free, require no credentials, and catch the most common agent-specific faults. The LLM leg is optional and should be treated as noise until proven reliable—judge variance is a real problem in LLM-as-judge setups.
How to instrument. Use the traces your runtime already emits. Do not build a custom tracing system—wrap your existing event stream. My JSONL-based approach works because my runtime already wrote one event per step. If yours does not, add a decorator or middleware that logs tool calls.
Where to store. A batch-compatible observability backend that supports trace-level scoring. I chose Langfuse for its REST API and per-trace scoring. Alternatives include LangSmith, Arize Phoenix, and OpenTelemetry collectors—each with trade-offs in latency, cost, and query capabilities. Upload in batches, track idempotency, and never block the serving path.
What not to build. No live dashboard (use the backend’s UI), no streaming upload (batch is simpler), no trajectory eval for agents that do not make sequential decisions (answer-level evals work fine). The boundary is clear: trajectory evals apply exactly where agency lives.
Tools and Frameworks for Agent Trajectory Observability
The source article focuses on Langfuse, but a vendor-neutral perspective is essential for production decisions. The AgentTelemetry benchmark (Balusu, 2026) provides a taxonomy of 14 fault types that any tool should detect. In practice, the choice depends on three criteria: whether you need a managed SaaS (LangSmith, Weights & Biases Prompts), open-source self-hosted (OpenTelemetry with a custom backend), or a lightweight REST-based system (Langfuse). My decision to use Langfuse was driven by its raw ingestion API—no SDK lock-in, idempotent batch uploads, and the ability to mirror the same REST shape across Python and TypeScript codebases.
For teams evaluating options, I recommend running a proof-of-concept with the AgentTelemetry span kinds on your own agent traces. Measure trace ingestion latency, storage cost per 10,000 spans, and the time required to write a custom judge like the one I built. The decision should be driven by these metrics, not by marketing claims.
Common Challenges and How to Solve Them
Sampling bias. A sampled online queue judges the traffic you get, not the failures you fear. For adversarial or rare-path trajectories, add targeted eval sets. The AgentTelemetry fault taxonomy can guide which edge cases to script.
The semantic-error blind spot. Trajectory observability is not a correctness oracle. An agent can walk a clean, non-redundant, well-abstained path and still produce a wrong answer—the tool sequence looks perfect, the content is subtly false. My deterministic checks (redundancy, abstention) catch process faults, not semantic ones; catching the latter still needs answer-level faithfulness scoring against cited sources. Trajectory and answer evals are complements, not substitutes. And the visibility is not free: every sampled trajectory is JSONL you store, upload, and process, so the sampling rate is a real cost dial, not a formality—which is why my default is 0 and coverage is opt-in per workflow.
The abstention trap. Rewarding “insufficient grounding” can teach the agent to give up. Gate abstention scoring on all retrievals being empty—not just low-confidence. My judge’s condition checks all(not hits for hits in retrieval_hits) before marking appropriate abstention.
AgentTelemetry is days old. Cite it as “just published,” not as established. Benchmark details may shift in the camera-ready version.
OpenAlex keyword match noise. The strict query for “agent trajectory” + observability catches some robotics and RL trajectory work. The trend direction is robust; the exact numbers are not. Use the direction, not the census.
Agent Trajectory Observability vs. Traditional Observability
Traditional observability focuses on system-level metrics, logs, and traces for monolithic or microservice applications. It tracks request latency, error rates, and CPU usage. Agent trajectory observability tracks LLM reasoning chains, tool selection decisions, multi-step orchestration paths, and semantic accuracy of outputs. The fundamental difference: traditional observability answers what went wrong at the infrastructure level; trajectory observability answers why the agent made that decision at the reasoning level.
This distinction matters for safety and alignment. A system that is perfectly healthy from a traditional observability perspective can still produce unsafe agent behavior—hallucinating a tool call, looping on a redundant retrieval, or failing to abstain when it should. Only trajectory observability catches these.
Best Practices for Production Deployment
- Instrument at the event level, not the request level. Capture every tool call, not just the final answer.
- Use deterministic checks as the first line of defense. Creds-free redundancy and abstention checks catch the majority of agent-specific faults.
- Batch upload with an idempotency ledger. Never block the serving path. Batch uploads are simpler and resumable.
- Sample with intent. Use a sampling rate that gives coverage without overwhelming storage. My default rate is 0—opt-in per workflow.
- Close the loop. Feed judge verdicts back into the same observability store as trace events. My v2 does this with Langfuse
score-createevents. - Know the boundary. Do not apply trajectory evals to stateless QA agents. The technique is only valuable where agents make sequential decisions.
FAQ
Q: What is agent trajectory observability? A: Agent trajectory observability is the practice of capturing, tracing, and analyzing the complete sequence of steps an AI agent takes to complete a task, including tool calls, LLM reasoning steps, intermediate outputs, and error states.
Q: How is agent trajectory observability different from traditional observability? A: Traditional observability focuses on system-level metrics, logs, and traces for monolithic or microservice applications, while agent trajectory observability tracks LLM reasoning chains, tool selection decisions, multi-step orchestration paths, and semantic accuracy of outputs.
Q: What tools are used for agent trajectory observability? A: Common tools include Langfuse (used in this post), LangSmith, Weights & Biases Prompts, Arize AI, Datadog with LLM observability integrations, and OpenTelemetry for custom backends. The right choice depends on your need for managed vs. self-hosted and your tolerance for SDK lock-in.
Q: Why is agent trajectory observability important for debugging AI agents? A: It allows developers to replay the exact sequence of steps an agent took, identify where reasoning broke down, detect hallucination patterns, and understand why an agent chose a particular tool or path—making debugging significantly faster than checking isolated logs.
Q: Can agent trajectory observability help with safety and alignment? A: Yes, by providing full visibility into an agent’s decision-making process, it helps detect unsafe tool usage, unexpected behavior patterns, and alignment failures, enabling teams to implement guardrails before deployment.
The ten other frontier capabilities I track—guardrails, autonomy ladders, critics, hierarchical planners, interactive feedback—all assume trajectory observability. You cannot build safe guardrails if you do not know what the agent actually called. You cannot evaluate an autonomy ladder if you only see the final rung. You cannot debug a critic that hallucinates its own feedback if you cannot replay the trajectory that led to that state.
Trajectory observability is the foundation. The research community is now formalizing it (AgentTelemetry benchmark, partial-observability-aware MARL, trajectory prediction under uncertainty). The tooling is trivial to wire up once you accept that the traces already exist—you just started reading them.
The correct answer is not enough. Judge the trajectory.