Agent harnesses can lose more correct answers than they gain, preprint finds

Adding agent software to a reasoning model can help it solve new questions while making its overall score worse. That is a central finding of an AlphaDiana preprint submitted to arXiv on 14 October 2026: in the authors’ tests, gains on previously unsolved questions could be outweighed by failures on questions the model answered correctly without tools.

The software around the model—its “harness”—organizes model calls, tools, skill instructions and memory. The researchers tested Qwen3.5-27B, Gemma-4-31B-IT and Kimi-K2.6 with OpenClaw, ZeroClaw and OpenCode across five benchmarks scored on final answers. Their reference, called Direct, gave the model one response per attempt without tools.

The clearest example came from Qwen on GPQA-Diamond. Each harness solved between eight and 13 questions that Direct missed, but failed on between 18 and 36 that Direct got right. Across all four setups, 175 of the 198 questions were answered correctly at least once, compared with 159 for Direct alone. That larger pool of solved questions did not translate into any individual harness beating the baseline.

AlphaDiana is designed to make those trade-offs inspectable. Rather than replacing each harness’s control loop with a common one, the system preserves its native behavior and links scores to execution traces and resource records. It also controls file reuse separately from memory retention, allowing researchers to change components under matched conditions.

Those component changes were no dependable fix in the reported experiments. Loading skill instructions improved accuracy in only two of 16 comparisons. Retained history did not reliably improve accuracy either. These findings concern the tested configurations, not every possible use of skills or memory.

The token figures expose another misleading improvement. On a shared set of Gemma/HLE-Verifiable tasks, OpenCode used 0.6% fewer output tokens than Direct but returned only 140 correct answers, against Direct’s 164. Once the researchers divided output from all attempts by the number of correct answers, output tokens per correct answer rose by 16.5%. The slightly shorter output had bought fewer successful answers.