Nonequilibrium Phases of Repulsive Self-Attention: Chaos, Attention Condensation, and Emergent Locality
We study the nonequilibrium dynamics of a minimal recurrent transformer with $N$ normalized tokens, $Q=K=I$, and a negative value map $V=-I$. Similarity-based attention selects nearby representations, while the negative value map drives tokens away from the selected field. This feedback can continually reorganize both the representation geometry and the attention network. For $
Reading in Plain English mode — toggle below to switch
We study the nonequilibrium dynamics of a minimal recurrent transformer with $N$ normalized tokens, $Q=K=I$, and a negative value map $V=-I$. Similarity-based attention selects nearby representations, while the negative value map drives tokens away from the selected field. This feedback can continually reorganize both the representation geometry and the attention network. For $
This research delves into the fundamental dynamics of the attention mechanism, revealing how negative value maps can lead to complex behaviors like chaos and emergent locality. Understanding these nonequilibrium phases is crucial for designing more stable and predictable transformer architectures, especially as models scale and exhibit increasingly complex emergent properties.