When an AI agent runs an end-to-end test in a real browser, the obvious next move is to ask it how the run went. It was there. It saw everything, and that's the problem with trusting its answer.

Self-grading is a known bias

The research that popularized using a model as a judge also cataloged how model judges go wrong. The one that matters most here is self-enhancement bias, where a model rates its own output more generously than a neutral party would.

An agent that has just fought its way through a long flow is the least neutral party available. Its closing summary describes what it set out to do, and a verdict can't be built from intent.

Split the roles by design

In our runner, one model drives the browser. A separate verifier model, which never took part in the run, makes one judgment per attempt. It gets two things:

• the test's expected outcome, written as itemized statements about the end state (including negative ones, like "no other records were modified")
• the recorded trajectory, meaning every action the agent took, every result that came back, and the agent's closing summary, labeled as the agent's own account

Its standing instruction is to judge only on evidence in that record. If nothing in the trajectory clearly shows an expectation was met, it isn't met, however confidently the summary claimed success.

The rules that keep the judge honest

A separate judge isn't enough on its own. We designed against specific failure modes.

Verbosity bias (rewarding long, confident prose) is the most dangerous one here, because an agent summary is exactly that. So the judge gives no weight to the summary's claims, only to recorded actions and results. We also built in an explicit asymmetry. Calling a run a pass when a core behavior broke is treated as much worse than a false alarm, so when the judge is in doubt about a core expectation, it marks that expectation missing.

Strictness like that can turn into noise, so there are pressure valves. An expectation the author marks [incidental] can never fail a run. A change that looks intentional, like a renamed button or an extra confirmation dialog, becomes a flag on a passing verdict. The flag records what was expected and what was seen, along with why it looks intended. A flag never excuses a core expectation that genuinely wasn't met.

If a failed run is retried on a stronger model, the retry gets its own fresh judgment, and nothing is averaged across attempts. The last rule is about admitting when there's no judge at all. If the verifier can't be reached, the run is recorded as failed with a note that the trajectory needs human review, and it's never escalated to the retry, since escalating a run nobody graded would manufacture confidence out of an outage.

What a verdict should hand you

A verdict should give you more than pass or fail. You get a reasoning summary, every expectation met, every expectation missing, any flags, and any extra actions the agent took that nobody asked for.

That's a real failed verdict from our QA environment. The test required one specific review cycle. The verifier's reasoning says the agent operated on a different one, so those expectations weren't met. Instead of investigating everything behind a bare red X, you'd check cycle selection first.

Where a model judge doesn't belong

None of this means you should replace coded assertions. If a property is machine-checkable behind a stable interface, a deterministic check is cheaper and faster. It's also perfectly repeatable, while a model judge is probabilistic. The judge earns its place on open-ended flows whose expected outcomes are written in plain language and whose UI won't hold still, which is where most end-to-end suites live.

The full post, LLM as a Judge Software Testing in Practice, covers bias and reliability in more depth and compares who grades a run under each testing approach.

Would you trust a model to grade your end-to-end runs? What would it need to show you first?