AWS SageMaker HyperPod 推出模型缓存，大幅减少推理冷启动
原标题：Reduce inference cold starts on Amazon SageMaker HyperPod with model caching
AI 摘要
AWS 为 Amazon SageMaker HyperPod 上的推理推出模型缓存功能，通过预加载模型权重和容器镜像到集群节点的本地 NVMe 存储，将推理 Pod 的冷启动时间从数十分钟缩短到数秒。该功能针对 DeepSeek-R1 等 600GB 以上大模型，解决自动扩缩容时因重复下载镜像和权重导致的响应延迟问题。缓存采用优先调度而非强制调度，未命中缓存的节点仍可回退到原始下载路径。
正文节选
Reduce inference cold starts on Amazon SageMaker HyperPod with model caching When you deploy a large language model (LLM) for inference on Amazon SageMaker HyperPod, there’s a gap between when you request a pod and when it’s ready to serve traffic. This gap is dominated by two sequential downloads: the inference server container image from Amazon Elastic Container Registry (Amazon ECR), and the model weights from your storage source, which can be Amazon Simple Storage Service (Amazon S3), Amazon