嵌入表太贵？面向小型与三值语言模型的16MB冻结词表
原标题：Embeddings Are Overpriced? A Frozen 16 MB Vocabulary for Small and Ternary Language Models
AI 摘要
该论文提出用一张冻结的16MB、512位二值码词表替代小语言模型中可训练的嵌入表，该词表由预训练嵌入经主成分投影和符号二值化一次性生成。在Gemma 3 270M同构模型上以相同数据对比训练，冻结词表模型在LAMBADA、HellaSwag、PIQA上表现持平或更好，仅ARC-Easy落后；配合三值BitNet主体后整模型仅46MB，笔记本CPU上生成速度达615-740 tokens/秒，约为float模型8-bit GGUF的5-6倍，内存仅为其七分之一。冻结词表在稀有词上损失低1.7-2.3 nats，训练后新增1万词仅付出3%代价，而可学习词表会崩溃43%。代码、权重、数据和评测结果均已开源发布。
正文节选
Embeddings Are Overpriced? A Frozen 16 MB Vocabulary for Small and Ternary Language Models Abstract In a small language model the vocabulary is the single largest part. In Gemma 3 270M the embedding table holds 168M of 268M parameters, and because quantisation methods leave the embedding in 16-bit, it is also most of the file once the rest of the model is compressed. We ask what that table actually buys. We replace it with one frozen table of 512-bit codes, 16 MB, with no trainable parameters, d