Triadic Linear Attention: Three-Dimensional Recurrent States for Long-Context Sequence Modeling
Recurrent neural networks (RNNs) compress the historical context into a memory state of fixed size, thus allowing for constant-time inference. The memory state size is a crucial factor in their performance, as exemplified by the strong performance and resurgence of linear attention, which extends the vector-valued hidden states of ordinary RNNs to matrix-valued hidden states. C
Reading in Plain English mode — toggle below to switch
Recurrent neural networks (RNNs) compress the historical context into a memory state of fixed size, thus allowing for constant-time inference. The memory state size is a crucial factor in their performance, as exemplified by the strong performance and resurgence of linear attention, which extends the vector-valued hidden states of ordinary RNNs to matrix-valued hidden states. C
This research introduces Triadic Linear Attention, a novel approach to recurrent neural networks that extends linear attention to a 3D tensor state. This could lead to more efficient and powerful sequence models, particularly for long-context scenarios, by improving memory state representation and constant-time inference capabilities.