What is a computer-use verification agent?

It is the inversion of the assistant-agent pattern. Instead of an agent that navigates other people's software on behalf of a human, a verification agent operates only your own UI to prove that each commit has not broken anything. A bounded agent explores the site while deterministic assertions decide pass or fail, so it stays reliable even though general computer-use agents are not.

Why do deterministic assertions decide the verdict instead of the LLM?

Because a verifier must not grade its own homework. The pure evaluate_route function judges HTTP status, horizontal overflow beyond 1px, expected text, required selectors, and filtered console errors — with zero flakiness by construction. The LLM triage is optional and env-gated; when active it may append an explanation to a failure but can never flip a verdict. On OSWorld 2.0 the best model hit only 20.6% binary completion, which is not CI-gate material.

How does the agent discover routes to verify?

A bounded breadth-first crawl (--discover --depth 2 --max-routes 25) takes a seed route, follows same-origin links while stripping fragments and query strings and dropping /api/, /auth/, and /_next/ paths, sorts the routes deterministically, and runs the full assertion suite on every discovered page. The max-routes cap prevents uncontrolled explosion and the deterministic sort makes the discovered set reproducible across runs.

How does the visual diff work?

The lane downscales both the current screenshot and a baseline to 64x64 grayscale and computes the mean absolute pixel difference against a default 2% threshold. At that resolution it catches layout collapse, missing chrome, and large misalignments while staying quiet on PNG-encoder noise and sub-pixel anti-aliasing — a deliberate trade against flakiness. Pillow is lazy-imported; if missing the lane emits a SKIP note and proceeds with the deterministic asserts.

How does this differ from OSWorld-style assistant agents?

OSWorld 2.0 measures agents operating unfamiliar software over long horizons — the best model reached just 20.6% binary completion across 108 tasks. A verification agent operates its own known surface, where the human already knows what correct looks like. The lane's deterministic asserts pass every known route when the UI is correct (10 of 10 in the last committed run); the gap between 20.6% and a clean sweep is exactly the gap between an agent on unfamiliar UIs and one on its own.

One post tagged with "osworld"

Computer-Use Agents for UI Verification: The Agent Explores, the Code Judges

July 6, 2026 · 17 min read

Vadim Nicolai

Senior Software Engineer

In late June 2026, the best computer-use agent failed four of five tasks on the new OSWorld 2.0 benchmark. Claude Opus 4.8 with max thinking hit just 20.6% binary completion on 108 long-horizon tasks, each requiring a median 1.6 hours of human interaction and roughly 318 tool calls. GPT-5.5 plateaued near 13%. (OSWorld 2.0, arXiv:2606.29537, submitted 2026-06-28.)

That reset changed what I think verification agents should be. Most researchers saw a challenge: build better agents that operate other people’s software. I saw the inversion. A computer-use agent that wanders any UI is too unreliable for unsupervised verification. But a bounded agent that operates only your own UI, with deterministic assertions as the decider and an LLM as an optional explainer, works today. It costs near zero per run, catches real regressions, and never grades its own homework.

This post documents the lane I built for my own site: just over 400 lines of Playwright-driven verification, a BFS crawler for route discovery, a 64×64 visual diff, and a closed loop where the page publishing the results is itself under verification. I’ll show how it draws from foundational research — OSWorld 2.0, the Rico, PIXELHELP, the Open Agent Architecture — and where those papers fall short for verification on broken UIs.