What is agent trajectory observability?

Agent trajectory observability is the practice of capturing, tracing, and analyzing the complete sequence of steps an AI agent takes to complete a task, including tool calls, LLM reasoning steps, intermediate outputs, and error states. It answers how an agent reached its answer, not just what it answered.

How is agent trajectory observability different from traditional observability?

Traditional observability focuses on system-level metrics, logs, and traces for monolithic or microservice applications, tracking request latency, error rates, and CPU usage. Agent trajectory observability instead tracks LLM reasoning chains, tool selection decisions, multi-step orchestration paths, and semantic accuracy of outputs — answering why the agent made a decision at the reasoning level.

How does a trajectory judge score an agent without credentials?

Two deterministic checks run with zero credentials: a redundancy check that flags the same tool called twice with the same query, corpus, and filters, and an appropriate-abstention check that rewards saying 'insufficient grounding' when all retrievals came back empty rather than hallucinating. An optional LLM leg refines the 0–1 score but never overrides the deterministic gates.

Why upload traces to Langfuse over raw REST instead of an SDK?

The uploader batches trace lines into Langfuse's public REST ingestion endpoint using stdlib urllib, with no SDK dependency, so the same REST shape mirrors across the Python and TypeScript codebases and both no-op cleanly when keys are unset. A per-directory ledger tracks uploaded lines, making reruns idempotent and partial uploads resumable, and uploads never block the serving path.

When should you not use trajectory evaluation?

Do not apply trajectory evals to stateless QA agents that do not make sequential decisions — answer-level evals work fine there. Trajectory evals earn their cost exactly where agency lives: agents with two or more tool calls in the typical path, agents that spawn sub-agents, or agents with failures that cannot be reproduced from the same input due to non-deterministic tool selection.

One post tagged with "agent-reliability"

Agent Trajectory Observability: Judge the Path, Not Just the Answer

July 6, 2026 · 14 min read

Vadim Nicolai

Senior Software Engineer

Two agents answer the same user query. Both return the identical string—correct, well-formatted, cited. An answer-level eval gives them both a perfect score, identical down to the decimal.

One agent made three redundant retrieval calls (same tool, same query, same corpus) before stumbling on the right source. The other called exactly the right tool once and answered. The answer-level eval cannot tell the difference. It never could.

The keys are in the trajectory.

I built a trajectory observability lane for my agents in three small pieces: the JSONL traces every workflow already emits but nobody reads, a judge that scores the tool-call sequence instead of the answer, and a Langfuse uploader written against the raw REST ingestion API—no SDK. Publication volumes indicate this is the moment: agent-observability research jumped sharply into 2026 (the phrase barely existed before), and the first dedicated fault-detection benchmark for agent observability was published this week.

This post is the full walkthrough: what trajectory observability is, why answer-level evals miss half the story, the three-module build, and how the research on partial observability validates the approach.