Pretraining Latent Information Feedback Transformers with Teacher Supervision
AI architecture language models (LMs) are feed-forward: deep-layer representations are never fed back to shallower layers, and the only pathway for information to flow downward across generation steps is the decoded token. This narrow channel forces models to recompute intermediate results and to discard alternative continuations. In this work, we remove this bottleneck during
Reading in Plain English mode — toggle below to switch
AI architecture language models (LMs) are feed-forward: deep-layer representations are never fed back to shallower layers, and the only pathway for information to flow downward across generation steps is the decoded token. This narrow channel forces models to recompute intermediate results and to discard alternative continuations. In this work, we remove this bottleneck during
This paper introduces a novel Transformer architecture that addresses a fundamental limitation of current LMs by enabling state propagation across generation steps. This could lead to more efficient and coherent long-sequence generation, reducing redundant computation and improving model performance.