OpenProblemBench turns to AI judges for unsolved math and physics

A preprint submitted on October 8, 2026 introduces OpenProblemBench, a test of AI systems built around 82 unresolved problems from mathematics and theoretical physics. The researchers, led in the author list by Zhiyi Li and Sihan Hu, use other AI models to assess the submissions without reference solutions.

The challenge is not simply to answer a difficult question with a known result. Each benchmark entry supplies research context, assumptions and previous progress. The authors say they selected problems whose proposed solutions permit comparatively clear checks of the crucial mathematical or computational claims—a way to make open-ended research more assessable.

Four evaluator models assess each submission for correctness, completeness and degree of progress. That setup makes the paper’s choice of metric important: it reports a “judged solve rate,” which describes the model evaluators’ assessments rather than establishing that researchers have confirmed new solutions.

Across seven evaluated configurations, the authors report that GPT-6-Astra had the highest mean judged solve rate, at 14.0%. The evaluated full-size open models scored between 5.5% and 6.7%, while Flash models scored between 2.4% and 3.7%. Those are author-reported results under this evaluation method.

The available evidence is the abstract, not the full paper. It does not spell out how the solve rate is calculated, how reliable the judges are or whether subject experts validated their decisions. Nor does it establish that any of the 82 problems has been resolved. The distinction is central: scoring an attempted solution is not the same as settling an open question.