Preprint tests what pretraining and midtraining contribute to learning from rewards

A new arXiv preprint by Chiwun Yang and Xiaoyu Li, submitted on 29 September 2026, asks what a model needs before reward feedback can teach it a task well. The authors start from a limitation: a reward can mark an answer as correct while leaving undetermined the computation needed to handle new inputs.

The paper examines two settings, sequential state computation and contextual memory. In both, the authors say they can characterize mechanisms that agree on every training reward yet demand different answers on held-out cases. They report that observations of the underlying source, which do not depend on the task reward, resolve that ambiguity. They also describe training paths in which a model first learns to predict the source and then adapts to rewards in the same parameters, and say they prove that prediction supplies execution or retrieval while rewards teach how to use it.

The experiments use pretrained Qwen2.5 checkpoints. According to the abstract, across eight worlds, sequential models trained with correct source and first-operation supervision reached 82.61% success, against 44.15% for a control using a private-random source. In a memory experiment, replay preserved retrieval during reward adaptation, and an independent eight-world confirmation reached 75.32% task success versus 49.86% after matched alternative-retrieval training. The paper also uses GSM8K and HotpotQA to separate accuracy when reward training starts, the gain afterward and final performance, but the abstract gives no scores for them.

What is confirmed and what is not: these figures are the authors' own, drawn from the abstract of a 160-page preprint. We have not read the full paper, and we found no independent coverage or replication, so this is a one-source item. The results apply to the constructed worlds and models tested, and the page does not claim they carry over to other models or applications.

Why it matters: the work separates two jobs often blurred together in reward-based training, supplying information and operations a model can use, and teaching it when to use them. If the pattern holds more widely, it would bear on how builders sequence pretraining, midtraining and reward adaptation. That remains for other researchers to test.

Revised 16 October 2026: Guth re-read the original sources and rewrote this article. The earlier version remains available as revision 1.