Exa's ATLAS benchmark says search agents are accurate but incomplete

Exa, the AI search company, announced a benchmark on Oct 8, 2026 that tries to measure something most search tests skip: whether an agent finds everything it should. ATLAS grades search agents on 547 "wide search" tasks, in which an agent must find every entity matching a set of conditions and fill in attributes for each.

The headline finding is Exa's own. It says the best agents it tested are accurate, with more than 90% of the entities they return being correct, but they miss a third or more of the entities that exist. Exa argues that gap matters for work such as building sales-lead lists or checking an investment target for adverse media, where one missed item can carry real cost.

What is new is less the idea than the method. Exa says existing tests are either largely memorized by frontier models, unlike real searches, or hard to grade precisely. It rejected LLM judges, saying they do not test completeness, and built a fixed answer key. Instead of human researchers, two builder agents, one running on ChatGPT and one on Claude, search independently for each task, and a separate verifier checks every row. Exa says each task took a median of 8 agent-hours, and that a sampled audit puts wrong answer-key values under 1%.

On Exa's runs, its own Agent Ultra got every attribute right about two thirds of the time, while the best frontier model, Opus 5.5, scored a little under 50 percent. Exa also says no agent costing under $1 per task passed 50%, and that swapping the search backend while holding the model fixed moved scores by up to 16%. Grading one system, it says, costs about 24 cents.

Several caveats apply. This is the only source we found: Exa designed the benchmark and sells a competing agent and search API, and the figures are self-reported. A secondary write-up we found repeats Exa's numbers rather than testing them. Exa says a blind human audit of the answer key is still under way, and that it will open-source the benchmark "in a few weeks," so outsiders cannot yet reproduce the results.

If the claims hold up, ATLAS would give buyers of research agents a way to compare completeness and cost, not just accuracy. Until independent groups run it, treat the rankings as Exa's account.

Revised 16 October 2026: Guth re-read the original sources and rewrote this article. The earlier version remains available as revision 1.