Netflix 详解内部 LLM 服务平台：Triton 与 vLLM 集成实践
原标题：Netflix Details its In-House LLM Serving Platform with Triton and vLLM
AI 摘要
Netflix 详细介绍了其内部 LLM 服务平台的架构，该平台基于 JVM 服务层，使用 Triton 进行模型管理和调度，并选择 vLLM 作为 GPU 推理引擎。平台面临模型大小、硬件需求和推理引擎快速演变的挑战，通过版本固定、扩展点和部署策略解决兼容性问题。Netflix 的经验表明，通用服务接口虽能抽象底层差异，但打包、兼容性控制和约束解码等工程工作仍需在各层进行。
正文节选
Netflix has described the production lessons behind bringing LLM inference into its internal serving platform, including the challenges of supporting different model sizes, hardware requirements, and rapidly evolving inference engines. The company’s account covers the architectural choices and operational work involved in running real-time and batch workloads across CPUs and GPUs, from model packaging and deployment to constrained decoding and version compatibility. The platform builds on Netfli