Thinkingbox benchmark grades business agents on the records they leave behind

Researchers have released a sandbox and benchmark that judge AI agents by what changes in a backend system, not by how convincing the conversation looks. The paper, "One Success Isn't Reliability," was submitted to arXiv on 20 August 2026 and last revised on 1 October 2026.

The authors say recent agent benchmarks increasingly use executable environments, but consequential work needs more than a plausible reply or a valid tool call. In their account, agents must gather missing information over several turns, follow domain policies, coordinate dependent tools and reach the correct persistent state without collateral effects.

Thinkingbox, as described by the authors, provides isolated tool sessions compatible with MCP, complete execution traces and evaluation of the final backend state. Its companion benchmark, Thinkingbox-bench, has 507 policy-conditioned workflows across retail, hospitality, auto insurance, neobank internal IT, and consulting IT and HR support. Each attempt is scored by task-specific executable checks designed to accept valid routes while rejecting wrong, missing or extra effects, and some tasks also check the final response.

The headline result is a reliability gap. The authors report that Claude Opus 5 falls from 66.50% pass@1 to 47.53% pass^20, and Kimi-K3 from 57.37% to 17.60%. The abstract does not define pass^20 in detail; the term appears to mean success across repeated attempts, in contrast with success on a single try. The authors also say many failed trials end cleanly after valid state-changing actions, so tidy-looking tool activity or a clean finish is a poor stand-in for completing the workflow.

What is confirmed and what is not: these figures come from the authors' own abstract for the paper's current version. We read only the abstract page, not the full paper, and found no independent replication or outside testing. Other write-ups we found summarize the same paper, and one reports different top-line numbers, which suggests figures may have changed between versions. This is therefore a one-source item.

The authors say they are releasing both Thinkingbox and Thinkingbox-bench. For teams building agents for business processes, the claim is a practical one: an evaluation that inspects the resulting system state, and repeats each task many times, can expose failures that a single successful run hides. Whether the benchmark's workflows predict real deployments is not something the abstract settles.

Revised 16 October 2026: Guth re-read the original sources and rewrote this article. The earlier version remains available as revision 1.