NVIDIA NIM 全栈优化让 Nemotron 3 Ultra 并发用户提升 2.5 倍

AI 摘要

NVIDIA 技术博客介绍其 NIM 2.0.12 全栈优化如何在 Nemotron 3 Ultra 上实现 2.5 倍并发用户提升。在 4xB200 硬件、64K 上下文、76% KV 复用、50 TPS/用户（20ms ITL）的 agentic 负载基准下，优化后吞吐从 718 tok/s 提升至 1,997 tok/s。优化来自精度与自动调优内核、张量并行、前缀与 Mamba 状态复用、调度批处理内存调优以及 MTP 推测解码等配置组合，而非独立开关叠加。文章还给出使用 AIPerf 回放流量、选择满足 SLO 的 Pareto 点及部署 NIM 2.0.12 的具体步骤。

正文节选

Deploying a large language model is only the first step toward production-ready serving. Production teams also need to serve as many concurrent users as possible on available GPU infrastructure while preserving the interactivity that keeps applications responsive. That tradeoff matters even more for agentic AI workloads, where prompts can be long, context can be reused across steps, and applications often stream extended responses back to users. NVIDIA NIM packages model- and GPU-aware serving c

发布时间：2026-09-11 00:55

抓取时间：2026-09-11 01:57

来源机构：NVIDIA