面向 LLM 的快速多项式超越函数
原标题：Fast Polynomial Transcendentals for LLMs
AI 摘要
该论文研究用短多项式程序替代 LLM 中 sigmoid、tanh、SiLU 等超越函数，以缓解 GPU 上 SFU 与矩阵/内存管线扩展不均衡造成的瓶颈。作者在 GB200 上对四个集成任务（dense SiLU、tanh-softcapped attention、sigmoid attention、routed-expert SwiGLU）用 BF16 三次或四次多项式替换原生实现，训练步吞吐分别提升 2.7%、2.9%、8.0%，sigmoid attention 前向提升 7.4%。论文还通过同 checkpoint 消融和配对预训练比较评估模型行为，并开源拟合工具与内核代码。
正文节选
Fast Polynomial Transcendentals for LLMs Abstract Graphics processing unit (GPU) generations scale matrix, special-function, and memory pipelines at different rates, so kernel bottlenecks move as hardware evolves. FlashAttention-4 exposed this imbalance inside attention on NVIDIA Blackwell. We test whether short polynomial programs can accelerate other special-function-unit (SFU) operations in large language models (LLMs). We first compare native PyTorch evaluation with packed fused multiply–add