矩估计的信任域框架

AI 摘要

本文提出了一种用于理解自适应矩估计（如Adam）在随机梯度优化中行为的信任域框架。该框架通过p阶矩约束限制每个权重的更新步长，推导出一系列基于二阶矩和归一化p阶矩的学习率机制，其中p=4时涉及峰度估计。实验表明，在GPT2-124M模型上，当信任域约束较弱时，四阶矩实现优势明显；随着约束增强，二阶矩实现更具竞争力。

正文节选

Computer Science > Machine Learning Title:A Trust-region Framework for Moment Estimation View PDF HTML (experimental) Abstract:In this paper, we develop a trust-region framework for understanding the behavior of adaptive moment estimation mechanisms, such as \textsc{Adam}, in stochastic gradient optimization. Specifically, in this framework, the magnitude of the update step for each individual weight is constrained within a trust-region governed by a moment constraint of order $p\in[

发布时间：2026-08-06 12:00

抓取时间：2026-08-06 20:49

来源机构：arXiv