数据可预测性决定Transformer权重规模增长规律

AI 摘要

该研究提出，训练后Transformer的权重规模可由双参数Weibull分布概括，其增长由语料库的二元条件熵（一种训练前可计算的统计量）预测。研究发现权重规模增长与可预测性边际的幂次呈仿射关系，且该关系在不同学习率下成立，并能提前预测训练结果。该定律在模型和层级别均适用，但跨语料库预测存在局限，代码语料因冗余性而偏离。

正文节选

Data Predictability Shapes Weibull Weight-Scale Growth in Transformer Training Abstract A trained transformer’s weight magnitudes can be summarized by a two-parameter Weibull distribution whose shape is stable across layers and models, so the scale carries most training-induced movement. What corpus property sets how much grows? Using the bigram conditional entropy , a training-free statistic computed before training, we find across controlled corruption families a learning-rate-conditioned law,

发布时间：2026-08-26 12:00

抓取时间：2026-08-26 12:02

来源机构：arXiv