Together AI 推出 LoLCATs：高效线性化 LLM 新方法
原标题：We're excited to introduce LoLCATs (Low-rank Linear Conversion via Attention Transfer), a new approach for quickly creating subquadratic LLMs from existing Transformers. Beyond simply accelerating mod
AI 摘要
Together AI 推出 LoLCATs 方法，通过注意力迁移将现有 Transformer LLM 快速转换为次二次复杂度模型，无需从头预训练。该方法在 Llama 3.1 全系列（8B/70B/405B）上首次实现线性化，仅用参数高效微调（<0.2%参数）和 4000 万训练 token，显著降低计算成本，同时提升零样本准确率，并匹配原始 Transformer 性能。
正文节选
We're excited to introduce LoLCATs (Low-rank Linear Conversion via Attention Transfer), a new approach for quickly creating subquadratic LLMs from existing Transformers. Beyond simply accelerating models, our focus is on creating fast models more efficiently, pushing the boundaries of AI development. Rather than invent and pretrain new architectures from scratch, LoLCATs builds on a recent playbook [1, 2, 3, 4] that simply (1) swaps out softmax attentions for efficient alternatives (e.g., linear