AWS SageMaker Inference 推出前缀感知路由降低 LLM 延迟
原标题：Reduce LLM latency with prefix-aware routing on Amazon SageMaker Inference
AI 摘要
AWS 在 Amazon SageMaker Inference 上推出 prefix-aware routing 路由策略，通过将具有相同提示前缀的请求持续发送到同一实例，使 KV 缓存得以复用。在 Llama 3.1 70B 的基准测试中，该策略将 P50 TTFT 降低最多 77%，吞吐量提升最多 16%，KV 缓存命中率从约 25% 提升至 80% 以上。该功能无需手动标记请求，并内置过载保护和扩缩容时的稳定性保障。
正文节选
Reduce LLM latency with prefix-aware routing on Amazon SageMaker Inference When you build an application on top of a large language model (LLM), the prompt you send to the model typically has two parts. There’s a fixed part that sets up context (instructions, reference documents, conversation history) and a variable part that contains the actual user input. Take a customer service bot as an example. Each request starts with the same block of text: “You are a support agent for AnyCompany. Here ar