FlashAttention-4：面向不对称硬件扩展的算法与内核流水线协同设计
原标题：FlashAttention-4: Algorithm and Kernel Pipelining Co-Design for Asymmetric Hardware Scaling
AI 摘要
Together AI 发布了 FlashAttention-4，一种针对 Blackwell 架构不对称硬件扩展的注意力内核优化算法。该算法通过软件流水线、多项式近似指数函数和 2-CTA MMA 等技术，在 B200 上达到 1605 TFLOPs/s（71% 利用率），比 cuDNN 9.13 快 1.3 倍，比 Triton 快 2.7 倍。
正文节选
Modern accelerators like Blackwell GPUs continue the trend of asymmetric hardware scaling, where tensor core throughput grows far faster than other resources such as shared memory bandwidth, special function units (SFUs) for transcendental operations like exponential, and general-purpose integer and floating-point ALUs. From the Hopper H100 to the Blackwell B200, for instance, BF16 tensor core throughput increases from 1 to 2.25 PFLOPs, while both the SFU count and shared memory bandwidth remain