Kalman Delta Networks: Uncertainty-aware Associative Memory
Linear attention is increasingly used in frontier language models for efficient long-context inference and constant-memory decoding. Its fixed-size recurrent memory, however, requires an online decision at each token: what to write and how strongly to overwrite existing associations before knowing which information future queries will require. Delta-rule models learn this stren
Reading in Plain English mode — toggle below to switch
Linear attention is increasingly used in frontier language models for efficient long-context inference and constant-memory decoding. Its fixed-size recurrent memory, however, requires an online decision at each token: what to write and how strongly to overwrite existing associations before knowing which information future queries will require. Delta-rule models learn this stren
This research introduces Kalman Delta Networks (KDNs), a novel approach to recurrent associative memory that explicitly models uncertainty using Kalman filters. For ML engineers, understanding KDNs can lead to developing more robust and efficient long-context models, particularly in scenarios requiring adaptive memory and uncertainty awareness.