Agent harnesses don't reliably make reasoning models more accurate, a new preprint finds

A preprint posted to arXiv on 14 October 2026 argues that wrapping a language model in an agent harness does not reliably make it better at reasoning problems, and can make it worse. The paper, from a team led by Zhanke Zhou at Hong Kong Baptist University's TMLR Group with collaborators at Stanford, Zuse Institute Berlin and elsewhere, also introduces AlphaDiana, a system for measuring why.

What the researchers did. A harness is the software around a model that organizes its calls, runs tools, loads skill instructions and keeps memory. AlphaDiana runs such agents on benchmark tasks while keeping each harness's own control loop intact, and it controls separately whether files and memory carry over between attempts. The full text names the test set: Qwen3.5-27B, Gemma-4-31B-IT and Kimi-K2.6, each paired with the OpenClaw, ZeroClaw and OpenCode harnesses and compared with a "Direct" baseline of one tool-free answer. Benchmarks included AIME 26, GPQA-Diamond and HLE-Verifiable, and two models were also run on two interactive benchmarks.

What they found. Harnesses "do not consistently improve" accuracy. Questions a model could not solve alone were sometimes solved with a harness, but those gains can be outweighed by losses on questions the model had answered correctly without tools. That happened for Qwen, whose scores fell under all three harnesses on all five answer-based benchmarks; other pairings differ. On GPQA-Diamond with Qwen, the four setups together solved 175 of 198 questions against 159 for Direct, yet every harness scored below Direct on its own. Skill guidance and retained history did not reliably help either; skill loading improved accuracy in only 2 of 16 comparisons.

Token use told a similar story. In one Gemma example on HLE-Verifiable, OpenCode used 0.6% fewer output tokens than Direct but produced 140 correct answers against 164. Low token entropy can also accompany wrong answers, so a model's apparent certainty, measured as low token entropy, is not evidence of correctness.

What is and is not established. These are the authors' results for three models, three harnesses and a set of benchmarks, and the arXiv page shows no peer review. They do not show that harnesses or skills are useless: ZeroClaw improved Gemma on all five answer-based benchmarks. We found no independent coverage or replication, so this article rests on the paper alone and every number comes from the authors' own reporting.

Why it matters. A natural question when choosing a harness is whether it can solve extra problems. This work suggests also counting what it breaks and what each correct answer costs in tokens. The authors recommend judging component changes by answer correctness and total output per correct answer.

Revised 16 October 2026: Guth re-read the original sources and rewrote this article. The earlier version remains available as revision 1.