PAR: Progress-Aligned Recurrence for Length Generalization in State Tracking
Looped language models can run more iterations at test time, but additional iterations need not help them solve longer problems. For sequential state tracking, final-answer supervision does not specify how far each iteration should advance, and repeated updates can change the scale of the recurrent state. We introduce Progress-Aligned Recurrence (PAR), which trains the predicti
Reading in Plain English mode — toggle below to switch
Looped language models can run more iterations at test time, but additional iterations need not help them solve longer problems. For sequential state tracking, final-answer supervision does not specify how far each iteration should advance, and repeated updates can change the scale of the recurrent state. We introduce Progress-Aligned Recurrence (PAR), which trains the predicti
This research addresses a critical limitation of current LLMs: their inability to generalize to much longer sequences than seen during training, especially for tasks requiring sequential state tracking. For ML engineers, understanding techniques like PAR is crucial for developing robust models that can handle complex, long-horizon problems without requiring prohibitively expensive training on extremely long sequences.