Together AI 推出缓存感知预填充-解码分离架构，长上下文推理提速达40%
原标题：Cache-aware prefill–decode disaggregation (CPD) for up to 40% faster long-context LLM serving
AI 摘要
Together AI 推出了缓存感知的预填充-解码分离（CPD）架构，通过按缓存命中率分离冷热工作负载，并利用分布式 KV 缓存，在长上下文推理中实现了高达 35-40% 的可持续吞吐量提升，并显著降低了首令牌延迟（TTFT）。该架构引入了预预填充节点、预填充节点和解码节点的三层角色划分，以及三级 KV 缓存层次，以优化混合真实流量下的服务性能。
正文节选
Serving long prompts doesn’t have to mean slow responses. At Together AI, we built cache-aware prefill–decode disaggregation (CPD), a serving architecture that purposely separates cold and warm workloads by cache hit rate, resulting in fast context reuse. By isolating heavy prefills and leveraging distributed KV cache, CPD delivers up to 40% higher sustainable throughput and significantly lower time-to-first-token (TTFT) for long-context inference — especially under mixed, real-world traffic. To