Distance generalization in transformers: why bother with positional encoding?
Out-of-distribution length generalization, namely to extrapolate a task from short to longer context, has been studied intensively for transformers. Here we focus on distance generalization, which probes performance when inter-token distances are changed between training and inference, while keeping a fixed context length. We construct two synthetic delay copy tasks, both invol
Reading in Plain English mode — toggle below to switch
Out-of-distribution length generalization, namely to extrapolate a task from short to longer context, has been studied intensively for transformers. Here we focus on distance generalization, which probes performance when inter-token distances are changed between training and inference, while keeping a fixed context length. We construct two synthetic delay copy tasks, both invol
This paper investigates a critical aspect of transformer performance: distance generalization, which is crucial for tasks requiring long-range dependencies. Understanding how positional encoding and data diversity affect this can lead to more robust and efficient model designs.