LLM架构新进展：KV共享、mHC与压缩注意力降低长上下文成本
原标题：Recent Developments in LLM Architectures: KV Sharing, mHC, and Compressed Attention
AI 摘要
本文回顾了近期开源大语言模型在长上下文效率方面的架构创新，重点分析了Gemma 4的KV共享与逐层嵌入、Laguna XS.2的逐层注意力预算、ZAYA1的压缩卷积注意力以及DeepSeek V4的mHC与压缩注意力。这些设计旨在减少KV缓存大小和内存开销，以支持更长的上下文和推理模型。文章指出，这些改动虽看似微小，但涉及Transformer块、残差流和注意力计算的复杂设计变化。
正文节选
Recent Developments in LLM Architectures: KV Sharing, mHC, and Compressed Attention From Gemma 4 to DeepSeek V4, How New Open-Weight LLMs Are Reducing Long-Context Costs After a short family break, I am excited to be back and catching up on a busy few weeks of open-weight LLM releases. The thing that stood out to me is how much newer architectures are focused on long-context efficiency. As reasoning models and agent workflows keep more tokens around (for longer), KV-cache size, memory traffic, a