Models still favored the subject after attention was redirected, preprint reports

Forcing grammar-focused attention heads to attend to a distractor noun did not make a language model’s syntactic predictions follow that distraction. The predictions remained more consistent with the actual subject, William Timkey and Tal Linzen report in a preprint submitted to arXiv on 13 October 2026.

The experiment did not produce a clean finding that these heads were irrelevant, either. Interventions on specialized syntax heads had a significantly stronger effect on syntactic predictions than interventions on randomly chosen heads—but only on one of the researchers’ two proposed measures. The authors describe the evidence for a causal role as “mixed to negative.”

The distinction under examination is between tracking grammar and driving behavior. As the authors explain, transformer language models develop attention heads that specialize in syntactic relationships, such as attending from a verb to its subject. Redirecting those heads tests whether changing that pattern also changes the model’s predictions. In the reported experiment, the subject remained the stronger influence even after the heads were pointed elsewhere. That is evidence against treating the attention pattern alone as an explanation of the prediction, not proof that the heads play no role.

The available evidence stops at the abstract page, which lists a four-page paper with two figures. It does not name the models or datasets, define the two measures or provide effect sizes. Without those details, this account cannot establish how broadly the result applies.