Shopify 推出 Gisting：将 LLM 系统提示压缩为学习令牌
原标题：Shopify Introduces Gisting: Compressing LLM System Prompts into Learned Tokens
AI 摘要
Shopify 工程团队推出了一种名为 Gisting 的新技术，可将长 LLM 系统提示压缩为少量学习的“要点”令牌，从而提高吞吐量并降低推理成本。在 Sidekick GraphQL 代理中，系统提示从约 6000 个令牌减少到 1500 个要点令牌，预测质量不变，中位首令牌时间从 438ms 降至 354ms，端到端延迟从 6.8s 降至 4.2s，吞吐量从 20.2 QPS 提升至 23.4 QPS，并减少了 GPU 分配。Gisting 基于 2022 年论文，通过教师-学生蒸馏学习要点令牌嵌入，无需修改模型核心权重，且可与前缀缓存等技术互补使用。
正文节选
Shopify's engineering introduced Gisting, a novel technique for compressing long LLM prompts into a smaller set of learned "gist" tokens, improving throughput and reducing inference cost. Spotify emphasizes that replacing lengthy text for concise gist tokens at inference time reduces end-to-end latency, drops infrastructure costs, and boosts token throughput without modifying the model's core weights. The company says that gisting reduced the Sidekick GraphQL agent’s system prompt from about 600