基于向量索引的输出嵌入加速LLM推理
原标题：Accelerating LLM Inference via Vector Index Based Output Embeddings
AI 摘要
该研究提出将大语言模型解码中的输出投影和top-k token选择重构为最大内积搜索（MIPS）问题，并使用基于HNSW的向量索引替代密集词汇投影，仅检索少量高得分token。在Gemma 3、Llama 3.2和Qwen 3模型上，该方法在CPU推理和批大小为1时，端到端吞吐量最高提升82%，同时保持生成质量。该方案无需重新训练，可无缝集成到现有模型中，适用于延迟敏感的小批量解码场景。
正文节选
Accelerating LLM Inference via Vector Index Based Output Embeddings Abstract Large output embedding matrices create a significant memory bandwidth bottleneck during autoregressive decoding, especially for compact LLMs with large multilingual vocabularies. We reformulate the output projection followed by top- token selection as a maximum inner product search over token embeddings and replace the dense vocabulary projection with an HNSW-based vector index. The resulting output head retrieves only