KVFetch：为KV缓存压缩缺失的另一半提供时间预取
原标题：KVFetch: Temporal Prefetching for the Missing Half of KV Cache Compression
AI 摘要
该论文指出当前 KV cache 压缩方法（基于分数的驱逐、摘要补偿、卸载召回）都只实现了按内容关联的寻址，缺少按位置顺序遍历的寻址通道，导致在 RAG、代码补全等需要逐字复制标识符的任务中出现「顺序遗忘」失败。作者提出 KVFetch，一个免训练、即插即用的框架，通过将低可预测性字面量降级到量化冷层、用单调读指针检测复制行为、并预取位置后继到固定大小热层槽位，为任意基于分数的压缩器打开时间召回通道。在 RULER-16K 等预算控制下，逐字复制从 0.8 提升到 78.4，13 任务平均提升 +8.4，且在无需顺序访问的 LongBench 上无额外开销。
正文节选
KVFetch: Temporal Prefetching for the Missing Half of KV Cache Compression Abstract As context windows scale to tens or hundreds of thousands of tokens, KV cache compression has become essential for efficient LLM inference. Existing methods fall into three families: score-based eviction, summary compensation, and offload-and-recall. Yet all three decide what to keep or recall by content relevance to the current query. We show this shared design is structurally incomplete. A cache supports two ac