现代LLM中的Engram：条件记忆的嵌入新方法

AI 摘要

Hugging Face 博客文章解释了现代LLM中的“engram”概念，即通过哈希多个token序列来创建额外嵌入，以缓解感知机学习局部序列信息的负担。文章指出engram本质上是嵌入，但通过哈希n-gram序列实现条件记忆，并讨论了哈希冲突、置信门控等技术细节。该方法由DeepSeek和美团等提出，旨在提升模型对常见序列的记忆能力，同时保留注意力机制处理罕见序列。

正文节选

Modern LLMs are built out of two primary components. Some kind of perceptron, like a MoE or a dense layer, and attention. Attention mixes in sequence information, and perceptrons do everything else. As you can probably imagine, that means perceptrons have to learn a lot of things and many of those things are quite hard. Engrams pick up the slack on some of these things, allowing the perceptrons to devote more of their parameters to the abstract witchcraft letting them capture lingustic concepts.

发布时间：—

抓取时间：2026-08-28 23:01

来源机构：Hugging Face