Epoch AI tests six models on its own work and finds they cannot yet replace staff

Epoch AI, a research group that tracks AI capabilities, says today's best models still cannot do its own work end to end. The conclusion comes from a new report, "Can AI automate Epoch?", published on 8 October 2026. This article rests on that report alone; the other coverage we found only restates it, so the findings have not been independently checked.

Epoch gave six models real assignments from its operations: graphic design, short data articles, interactive data explorers, research on AI data centres, and designing a pilot experiment. The launch suite has 11 tasks. Each model worked in its own coding-agent software with the highest reasoning setting, with access to relevant files and its own Google account, and without human help. Each output was then graded by a single person against the organisation's internal standards.

Epoch says Claude Fable 5.1 and GPT-6 Astra broadly led, and were reliable on well-defined work such as coding and data analysis. It says computer use was better than it expected: models that once failed to move an article to Substack now manage it. Epoch dropped one design task from the suite because Fable 5.1 produced work it judged to be of its own quality, and it now largely automates that step.

The shortfalls came in judgment. By Epoch's account, the models missed unwritten conventions such as visual style and what its audience cares about, and sometimes produced denser material than it wants. In the research-design task, GPT-6 Astra built a pilot experiment that gave the models it tested only 4,096 output tokens, which cut off 61 of 280 responses before they could answer. Epoch says Astra raised the limit but still described the resulting score changes as a real finding. Open-weight models trailed further: Epoch says Kimi K3 built a data article on a filter that counted every arXiv paper, not just physics papers. Three of six models also picked the same polling topic.

Epoch lists its own limits. Each model ran each task once, each output was graded by a single person, and grading was partly subjective, models used different software setups, and the tasks are specific to Epoch. It says the scores are noisy and the results are not a general measure of AI at work.

The report matters less for its rankings than for its method. Epoch argues that benchmarks built on easily scored answers can miss whether a system picks up an employer's standards or knows when a result is flawed. Epoch's conclusion is that AI cannot yet replace workers, at least not at Epoch, and it plans to add results as new models arrive.

Revised 16 October 2026: Guth re-read the original sources and rewrote this article. The earlier version remains available as revision 1.