OasisKV：利用前瞻稀疏预取将解码 KV 缓存扩展至 HBM 之外
原标题：OasisKV: Scaling In-Decode KV Cache Beyond HBM with Lookahead Sparse Prefetching
AI 摘要
OasisKV 是一个面向大语言模型推理的内存中心系统设计，通过将 KV 缓存与 HBM 解耦来缓解 HBM 容量压力。它利用推测解码生成的 lookahead tokens 预测未来重要 token，并采用高效的注意力后台流水线从更高容量内存层级预取相关 KV 块。基于 vLLM 的实现显示，在 2048 token KV 预算下精度损失不超过 0.7 点，推理吞吐量提升 1.69 倍，多 GPU 长上下文场景下提升 2.1 倍。
正文节选
OasisKV: Scaling In-Decode KV Cache Beyond HBM with Lookahead Sparse Prefetching Abstract Large language model (LLM) inference serving is increasingly constrained by memory rather than compute. As long-context and long-form reasoning workloads become more prevalent, the key-value (KV) cache dominates both memory footprint and memory traffic during LLM token generation, i.e., decode. In particular, HBM capacity has become a scarce and costly resource that heavily limits inference batch size and s