Learning Length-Extrapolatable Recurrent Models
Recurrent models provide a natural path to long-context modeling, yet models trained with backpropagation through time (BPTT) often fail beyond their training horizon. Classical analyses emphasize gradients that vanish or explode along temporal paths. However, dense per-token losses can still train a shared recurrent rule despite severe decay, showing that decay alone does not
Reading in Plain English mode — toggle below to switch
Recurrent models provide a natural path to long-context modeling, yet models trained with backpropagation through time (BPTT) often fail beyond their training horizon. Classical analyses emphasize gradients that vanish or explode along temporal paths. However, dense per-token losses can still train a shared recurrent rule despite severe decay, showing that decay alone does not
This paper addresses a fundamental limitation of recurrent models in handling long contexts, a critical area for many sequence-to-sequence tasks. Understanding and applying techniques like Credit Stabilization through Time (CST) can significantly improve the performance and applicability of RNNs in real-world applications, especially where long-range dependencies are crucial.