2 posts tagged with "agent"

Survivorship bias is supposed to flatter a backtest. A survivor-only universe deletes the names that died along the way. Every number computed on it should therefore come out looking better than the truth. That is the textbook direction — and for this board, the textbooks had it backwards.

The measurement that broke the assumption came from a 10-minute autonomous research loop. It ran the previous evening and logged the result as a measurement only: no lane, constant, module, or gate default was changed.

The loop re-screened its own universe. The survivor-only reference — a single active=true snapshot of Polygon's ticker list — had been used to type every name on all 236 point-in-time dates. That reference produced a benchmark that was too low.

Readmitting every name the gate had silently excluded moved the equal-weighted screened universe from +5.64 to +6.54 bps at k=1, and from +27.34 to +29.07 bps at k=5.

Read that table twice.

The bias did not flatter the backtest. It censored the names that made the backtest look worse. The reason is structural, not mystical: this panel never observes a delisting as a return. There is no −100% row to be spared.

Removing names did not remove disasters. It removed a type of name — and that type was exactly what the extreme-return lanes were looking for.

Every team running an agent fleet has the same blind spot. Observability platforms—MLflow, Langfuse, home-grown OpenTelemetry—capture execution traces beautifully. They show you what the agent did. They say almost nothing about whether it did it well. So a developer opens the trace viewer, scrolls through a few hundred spans, and tries to eyeball a systemic failure out of thousands of runs. The research alternative is worse: hand-built error taxonomies that take weeks to annotate and go stale the moment the agent changes. What both approaches lack is automated multi-level agent evaluation—judgment of the trajectory itself, not just a record of it.

Agentic CLEAR: Automating Multi-Level Evaluation of LLM Agents, by Yehudai, Eden, and Shmueli-Scheuer (2026) at IBM Research, attacks exactly this gap. It is an open-source Python package—pip install clear-eval—that reads raw agent traces and produces data-driven evaluation at three levels of granularity, surfaces recurring failure patterns without a predefined taxonomy, and renders the whole thing in an interactive dashboard. It reports up to 0.890 AUC for predicting trajectory success in a fully reference-less setting. This post walks through what the paper actually does, then shows how I wired the same multi-level shape into a 45-graph production fleet as an "autonomy gate"—the component that turns a human approval interrupt into a machine one.