Study Finds Many Rollouts Needed to Identify LLM Decision Cruxes

The arXiv preprint titled *Can LLMs Anticipate Their Cruxes?* was submitted on 14 Oct 2026 by David Reber, James Reber, Ari Holtzman and Victor Veitch. The authors investigate how to surface the underlying reasons that drive a language model’s recommendation in a decision‑making context. They note that while a model can embed rationales in its generated text, the volume of output makes direct inspection impractical for real‑world use.

The paper defines a “crux” as a sub‑question on which a model’s rollouts disagree, and whose answer could flip the final conclusion, making it a property of the rollout distribution rather than any single answer. Identifying such cruxes is presented as a way to make model‑based recommendations more transparent and trustworthy.

The authors evaluate three strategies of increasing computational cost. First, they ask the model to predict, a priori, which cruxes will matter for a decision. Second, they let the model generate a response and then extract cruxes that emerged during its reasoning. Third, they generate many responses and compare reasoning across rollouts to isolate divergent sub‑questions. To keep this many‑rollout approach tractable, they use cheap calls to a reranker, which they estimate requires about 86 times less compute than a full judge model.

Extensive experiments across domains and model families show that the inexpensive options fall short. The a‑priori assessment of cruxes has little correlation with the factors identified after generation, indicating poor predictive power. Using a single rollout improves matters but still fails to capture the full set of influential cruxes. Consequently, the authors conclude that for high‑stakes decisions the many‑rollout procedure is unavoidable.

For AI builders, the results imply that systems relying on LLMs for critical recommendations must budget for multiple rollouts and reranking to surface true decision‑making cruxes. The cheap reranker technique offers a practical path, reducing the compute burden while still enabling many‑rollout analysis. Incorporating scalable rollout pipelines and efficient rerankers therefore becomes essential for reliable, high‑stakes AI applications.