FlashAttention-3 发布：利用 Hopper GPU 新特性实现 1.5-2 倍加速

AI 摘要

Together AI 发布了 FlashAttention-3，通过利用 Hopper GPU 的异步特性（如 WGMMA、TMA 和 FP8 低精度）来加速注意力计算，相比 FlashAttention-2 在 FP16 下速度提升 1.5-2 倍，达到 740 TFLOPS（H100 理论峰值的 75%），FP8 下接近 1.2 PFLOPS。该技术提高了 GPU 利用率、支持更低精度且保持准确性，并支持更长上下文，代码和论文已公开。

正文节选

Attention, as a core layer of the ubiquitous Transformer architecture, is a bottleneck for large language models and long-context applications. FlashAttention (and FlashAttention-2) pioneered an approach to speed up attention on GPUs by minimizing memory reads/writes, and is now used by most libraries to accelerate Transformer training and inference. This has contributed to a massive increase in LLM context length in the last two years, from 2-4K (GPT-3, OPT) to 128K (GPT-4), or even 1M (Llama 3

发布时间：—

抓取时间：2026-08-02 00:28

来源机构：Together AI