AlphaDiana Measurement System Evaluates Reasoning Agents and Harnesses

AlphaDiana is a newly released measurement system designed to evaluate reasoning agents that combine large language models with execution harnesses. The system was submitted to arXiv on 14 Oct 2026 by Zhanke Zhou and eighteen co‑authors, and it targets agents that pair models with harnesses that organize calls, execute tools, load skill instructions, and retain history in memory.

AlphaDiana offers modular interfaces for datasets, agents, environments, and scorers, allowing researchers to keep the native control loops of existing harnesses while independently managing file reuse and memory retention. This design supports both macro‑level analyses of whole agents and micro‑level interventions on components such as tools, skills, or retained history under matched experimental conditions.

In its initial experiments the authors evaluated three open‑source reasoning models and three open‑source harnesses across five answer‑based benchmarks, and two of the models were also run on two interactive benchmarks. The results show that harnesses do not consistently improve reasoning accuracy; gains on questions the model could not solve alone can be offset by losses on questions it answered correctly without tools. Likewise, targeted interventions such as adding skill guidance or preserving execution history failed to reliably raise accuracy, indicating that neither component alone guarantees better performance.

The study also highlights that token‑level behavior must be examined alongside correctness, because low token entropy can accompany wrong answers and reduced output per execution may increase tokens spent per correct answer when overall accuracy drops. AlphaDiana links answer correctness and execution status to detailed traces and resource records, enabling systematic comparison of agents and diagnosis of failure modes under shared task and resource policies.

The study shows that adding harnesses or component interventions does not reliably improve accuracy, highlighting the need for systematic measurement of agent performance. By linking correctness to execution traces and resource use, AlphaDiana enables developers to compare agents and diagnose failure modes under shared policies.