New benchmark asks whether AI judges can tell a finished computer task from a lookalike

A ten-author preprint posted to arXiv on 8 October 2026 describes AgentHorizon, a test not for computer-use agents themselves but for the automated judges that decide whether those agents succeeded. The authors say such judges are increasingly used to grade agents during training and evaluation without a person checking the work, yet their reliability on long tasks that span several applications is unclear.

The problem they target is that a long record of screenshots and actions, called a trajectory, can look complete while breaking a constraint in the instruction or leaving an unwanted side effect.

AgentHorizon contains 1,373 tasks, each an instruction paired with a trajectory, drawn from 166 hours of human-recorded activity on three operating systems. According to the authors, the key design choice is to record trajectories for closely related instructions and then swap the instructions to create negative cases. A judge must therefore separate a trajectory that satisfies the request from one that completed a similar but incompatible request.

The benchmark ships in three splits: a frontier split (AH), a simplified split (AH-S) and a development split (AH-D). The authors tested eleven judges two ways, by passing in whole trajectories of up to 300 screenshots and actions, and by running the models as coding agents inside five agent harnesses.

The headline result is that the best agentic judge, GPT-5.5, reached 80.9% balanced accuracy on the AH split. The authors also report that tool use improved some models but made open-weight models worse, and that judges differed drastically in how well they accepted valid trajectories versus rejected failed ones. They conclude that judges must find evidence of completion buried in long histories.

What is confirmed here is limited to what the authors state on the arXiv abstract page. The figures are the authors' own and have not been independently reproduced. The abstract does not give scores for the other judges, name the models in each group, or say how the data and code can be obtained, and we have not read the full paper. It is a preprint. We found no independent coverage or replication of this specific paper, so this item rests on a single source.

It matters because the numbers an agent reports about itself are only as good as the judge scoring it. If a judge is lenient with plausible-looking but wrong runs, training signals and leaderboard results can be inflated. A benchmark built from near-miss pairs gives developers a way to measure that weakness separately from task execution.

Revised 16 October 2026: Guth re-read the original sources and rewrote this article. The earlier version remains available as revision 1.