Deep Fusion Memory: A Chunk-Level Parametric Memory for Large Language Models
Large language models need effective ways to acquire and use external knowledge beyond what they retain from pretraining. Continued pretraining risks catastrophic forgetting, while retrieval at inference time can add substantial overhead. External parametric memory avoids online retrieval, but constructing its training data requires extensive offline search, and output-level in
Reading in Plain English mode — toggle below to switch
Large language models need effective ways to acquire and use external knowledge beyond what they retain from pretraining. Continued pretraining risks catastrophic forgetting, while retrieval at inference time can add substantial overhead. External parametric memory avoids online retrieval, but constructing its training data requires extensive offline search, and output-level in
This research introduces a novel parametric memory architecture that aims to improve LLM knowledge acquisition and utilization without the overhead of online retrieval or the risks of catastrophic forgetting from continued pretraining. For ML engineers, understanding such memory mechanisms is crucial for building more efficient and knowledgeable LLMs, especially in applications requiring extensive external knowledge.