AWS 推出 SageMaker HyperPod 分层 KV 缓存，提升 LLM 推理性能
原标题：Tiered KV cache for large LLMs on Amazon SageMaker HyperPod with Curvine
AI 摘要
AWS 在 SageMaker HyperPod 上推出分层 KV 缓存架构，结合 Curvine 分布式缓存文件系统，将 KV 缓存从 GPU 扩展到 CPU 和共享 NVMe 池，实现跨副本缓存复用。测试显示，该方案可将跨 Pod 缓存命中率提升至 100%，TTFT 改善最高达 2.7 倍，并允许在更低成本的 G6e 实例上运行原本需要 P5 实例的工作负载。
正文节选
Tiered KV cache for large LLMs on Amazon SageMaker HyperPod with Curvine Running large language model (LLM) inference at scale typically forces a KV cache trade-off: you either pay for oversized GPU instances to accommodate a growing KV cache, or you accept slow time-to-first-token (TTFT) as identical prompts get recomputed on every request. For teams deploying a broad catalog of publicly available foundation models (FMs), such as Qwen, Llama, DeepSeek, and others, across per-business-line endpo