OpenProblemBench tests AI on 82 unsolved math and physics questions

A research team has proposed a new way to measure whether AI can contribute to science: ask it to work on questions nobody has answered. The preprint, posted to arXiv on 8 October 2026 by Zhiyi Li, Sihan Hu and colleagues, introduces OpenProblemBench, a set of 82 unresolved problems drawn from the mathematics and theoretical physics literature.

According to the authors, each problem comes with the research context, assumptions and prior progress needed to investigate it. They say they chose problems whose proposed solutions allow comparatively clear checks of their decisive mathematical or computational claims. That matters because open problems have no answer key to grade against.

The benchmark's workaround is to use models as judges. Four evaluator models independently assess each submission for correctness, completeness and degree of progress, without reference solutions. The headline number is therefore a "judged solve rate", a score from model graders rather than a count of problems confirmed solved by mathematicians or physicists.

Across seven evaluated configurations, the authors report that GPT-6-Astra reached the highest mean judged solve rate, 14.0%. The evaluated full-size open models scored 5.5% to 6.7%, and Flash models 2.4% to 3.7%. The authors also say that stronger outcomes in their case comparisons were connected to changes in problem representation, general arguments that extend beyond finite evidence, and proofs of the steps needed to complete a solution.

What is new is the focus. Most benchmarks test established knowledge; this one is built around questions arising from the research literature, which the authors describe as a setting for studying AI's capabilities and limitations as a contributor to foundational theoretical science.

What is confirmed is limited to what the paper's abstract states. We found no independent second source covering this paper, so this item rests on the abstract alone, and we have not reviewed the full text. The scores are author-reported and have not been independently replicated. It is unclear how reliable the model judges are, how partial progress is scored, or whether any problem was actually resolved; the abstract does not claim that any was.

The benchmark gives researchers a shared, if imperfect, yardstick for a harder question than exam-style tests. Its value will depend on whether the grading holds up under expert scrutiny.

Revised 16 October 2026: Guth re-read the original sources and rewrote this article. The earlier version remains available as revision 1.