EngramEdit: Decoupled Knowledge Updates in LLMs
EngramEdit is a novel method for updating factual knowledge in Large Language Models (LLMs) that use conditional memory architectures, like DeepSeek Engram. It allows for precise modification of specific facts without retraining the entire model, addressing a key challenge in maintaining and evolving LLM knowledge bases.
Reading in Plain English mode — toggle below to switch
EngramEdit is a novel method for updating factual knowledge in Large Language Models (LLMs) that use conditional memory architectures, like DeepSeek Engram. It allows for precise modification of specific facts without retraining the entire model, addressing a key challenge in maintaining and evolving LLM knowledge bases.
Imagine an LLM's knowledge as a vast library. Traditional updates mean rewriting entire books (retraining), which is inefficient. EngramEdit is like updating a specific entry in the library's card catalog (conditional memory) to point to new information, without disturbing other books or the librarian's general knowledge of how to find things.
How EngramEdit Works
- 11. Identify Target: Determine the specific factual knowledge to be updated in the LLM.
- 22. Compute Target Memory: EngramEdit calculates ideal memory representations that would make the model predict the new fact across various ways of asking the question.
- 33. Jointly Update Embeddings: It then adjusts the shared n-gram embeddings in the conditional memory to align with these new target representations.
- 44. Preserve Unrelated Knowledge: During updates, a penalty is applied to changes in frequently used embeddings, minimizing unintended side effects on other knowledge.
L = Lₑdit + λ * L_preserve
The total loss $L$ is a combination of an editing loss $L_{edit}$ (ensuring the new fact is learned) and a preservation loss $L_{preserve}$ (preventing changes to unrelated knowledge), weighted by a hyperparameter $λ$.
If an LLM incorrectly states 'The capital of France is Berlin,' EngramEdit can update the conditional memory associated with 'capital of France' to correctly output 'Paris,' without affecting its knowledge about German cities or other geographical facts.
EngramEdit transforms conditional memory into an editable interface, enabling precise, decoupled factual knowledge updates in LLMs while preserving general capabilities.