AgentHorizon puts AI judges to the near-miss test

The strongest agentic judge tested in AgentHorizon, a benchmark introduced in an October 8 preprint, reached 80.9% balanced accuracy, according to its authors. The target is not the AI operating a computer, but the AI asked to decide whether the work was done correctly. The researchers identify GPT-5.5 as their best-performing agentic judge on the benchmark’s frontier split, AH.

AgentHorizon’s central trick is an instruction swap. Researchers record computer activity for closely related requests, then pair a recording with an incompatible instruction. A judge must spot that the work satisfies a different request, even if the sequence looks convincingly complete. That tests something more demanding than recognizing an obvious failure: checking the work against the actual assignment.

The benchmark contains 1,373 instruction-and-recording pairs drawn from 166 hours of human-recorded activity across three operating systems. These histories, which the paper calls trajectories, consist of screenshots and actions. The authors argue that apparent completion can conceal a broken constraint or an unwanted side effect, especially in long tasks spanning multiple applications.

They evaluated eleven judges in two modes: feeding them whole trajectories containing up to 300 screenshots and actions, and running them as coding agents across five agent harnesses. Tool use was not a uniform improvement. The authors report gains for some models but worse performance for open-weight models, alongside substantial differences in judges’ ability to accept successful runs and reject failed ones.

Those findings remain author-reported preprint results. The captured abstract supplies no model-by-model score table or measurements of the tool-use gains and losses. The 80.9% result applies specifically to the AH split; it should not be read as a general success rate for judging computer work.