缩放定律的细致探讨：从理论到实践

AI 摘要

本文回顾了深度学习中的缩放定律（scaling laws）研究，从早期理论到Kaplan等人的实证工作，指出损失随模型规模、数据和计算量呈幂律下降，并可用于预测大模型训练需求。文章强调了缩放定律在计算资源分配中的实用价值，并讨论了不同阶段（小数据、幂律区、不可约误差区）的行为。

正文节选

Scaling laws are one of the most critical empirical findings in deep learning. The observation is simple in form: the training loss $L$ decreases predictably as we scale up model size $N$, dataset size $D$, and compute $C$, following a power-law curve, which appears as a straight line on a log-log plot. We can view scaling laws as a framework for describing the relationship between compute, loss, model size and data; at its core, it is about how to allocate precious compute optimally between $N$

发布时间：2026-06-24 08:00

抓取时间：2026-08-02 00:26

来源机构：Lilian Weng