AWS 推出 SageMaker HyperPod 推理网关
原标题：Introducing Amazon SageMaker HyperPod Inference Gateway
AI 摘要
AWS 发布 Amazon SageMaker HyperPod Inference Gateway，这是一个以 EKS 托管插件形式部署的 Kubernetes 原生、GPU 感知推理路由系统。它通过 Envoy Gateway、Body-Based Router 和 Endpoint Picker 三层组件，利用 KV 缓存利用率、队列深度、LoRA 适配器驻留等实时指标进行加权打分路由，宣称可将首 token 延迟降低最多 82%，并减少 GPU 浪费。该网关支持多模型路由、LoRA 适配器路由、优雅故障恢复和内置可观测性，且无需修改应用代码。
正文节选
Introducing Amazon SageMaker HyperPod Inference Gateway Eliminate GPU waste. Reduce first-token latency by up to 82%. Install one Kubernetes-native addon with zero application changes. The problem: Naive routing wastes your most expensive resource Running large language models (LLMs) at scale on GPU clusters is expensive. The default Kubernetes load balancers are making it worse. Round-robin and least-connections algorithms have no visibility into what’s happening inside your GPUs: which pods ha