Epoch’s automation tests find research judgment lagging behind coding skill

Writing code was not the main obstacle in Epoch’s test of whether AI could automate its work. In a report dated October 8, 2026, the organization says its leading models handled coding and computational analysis reliably but fell short on open-ended work requiring research judgment or an understanding of its standards.

One research-design assignment makes the distinction concrete. GPT-6 Astra proposed studying whether AI agents fail to improve because they gather poor evidence or fail to use evidence they already have. Its pilot involved models trying to discover a puzzle’s hidden rule by choosing inputs and observing the results.

But Astra’s setup allowed only 4,096 output tokens while the tested models were choosing an input. That cut off 61 of 280 responses before they named one. Those responses still counted as attempts, leaving the models with less evidence than intended. Astra spotted the problem and raised the limit, improving scores—but still framed the changes as a finding about the budget for choosing inputs. Epoch argues that this confused incomplete responses with meaningful differences in how models used that budget.

The evaluation covered six models and, at launch, 11 tasks spanning graphic design, short data articles, interactive data explorers, data-center research and research design. Epoch supplied relevant resources, workspaces and Google accounts, used the highest available reasoning settings, and let the models proceed without further intervention. A single human grader reviewed each output against rubrics combining objective requirements and subjective judgments.

Claude Fable 5.1 and GPT-6 Astra broadly led the evaluation. There were practical successes: Epoch says it removed a relatively straightforward chart-restyling task after Fable produced work that met its quality bar. The organization now largely automates that step. That is a narrower achievement than independently producing a finished research project, but a useful one.

Elsewhere, supplying examples was not enough. Epoch says models missed implicit conventions about its visual style and audience. Fable’s graphics and writing could be much denser than the organization wanted; Astra selected an article topic Epoch judged too specialized for its readers. These were failures to apply house standards, not simply failures to execute instructions or calculate results.

The findings are specific to this test. Each model attempted each task once, grading was partly subjective, and models ran in different software setups. Epoch cautions that the scores are noisy and the tasks do not represent work in general. Its examples document a gap between completing well-defined steps and delivering work that meets the organization’s standards—not a general verdict on which jobs AI can automate.