REST adds thought-level losses to latent reasoning training, with gains reported by its authors

Researchers Fahd Seddik and Fatemeh Fard have posted an arXiv preprint, submitted on 28 September 2026, proposing a way to train language models that reason in continuous space rather than in decoded text. This account rests on that single abstract; we found no independent coverage or replication.

The setting is "latent" reasoning, where a model recurs on its own hidden states or passes them between agents. According to the authors, training in such systems usually supervises only the cross-entropy of the final decoded answer, leaving the intermediate thought unconstrained. They say theoretical and empirical analyses confirm four failures of that approach, including thoughts collapsing across distinct questions and retaining irrelevant information.

The new element is REST, short for REpresentation-Supervised Thoughts. It turns four properties of a valid thought representation, namely causality, minimality, separability and stability, into differentiable losses added to cross-entropy. The authors apply it to single-agent and multi-agent latent systems and say it needs no architectural changes and adds no parameters at inference.

On results, the authors report that across seven benchmarks in mathematics, science, medicine and code generation, using the same training data, compute and latent budget, REST raised accuracy over cross-entropy-only training by up to 7.5 percentage points and improved convergence on a final answer by up to 30 percent. They also say REST thoughts encode more of what is needed for a correct answer, and that decoding them better recovers the agent's intended output, which they argue makes latent communication easier to interpret.

What remains uncertain is considerable. These figures are the authors' own maximum gains, the abstract gives no per-benchmark numbers, and it does not define how convergence is measured. We have not read the full paper, and the work has not, as far as we can tell, been independently reproduced. The preprint status also means it may not have been peer reviewed.

It matters because multi-agent systems that exchange hidden states are hard to inspect, and a training objective that makes those states more decodable would address a real gap, if the claims hold up. For now, treat the results as a promising claim from one research group rather than an established finding.

Revised 16 October 2026: Guth re-read the original sources and rewrote this article. The earlier version remains available as revision 1.