相关性正在破坏你的梯度下降
原标题：Correlations Are Ruining Your Gradient Descent
AI 摘要
该论文研究自然梯度下降中由层输入数据相关性产生的右因子，指出其本质是基变换而非曲率问题，可移出优化器并放入网络内部。作者提出三种等价实现形式：梯度预条件器、前向传播中的去相关线性变换，以及仅依赖局部信号、全部更新为Hebbian的循环动力学。该方法无需矩阵求逆或分解，仅需单个突触可用信息，可在神经形态硬件等分布式物理约束系统中应用，并在卷积和残差网络上验证了有效性，同时改进了多种近似生物合理的信用分配方法。
正文节选
Correlations Are Ruining Your Gradient Descent Abstract Natural gradient descent, and the preconditioned optimizers which have followed from it, correct gradient descent by estimating curvature and applying its inverse to the gradient. For any single layer of a deep neural network, this correction can be shown to approximately separate into two factors: one which accounts for correlations in the gradient signal, and one which accounts for correlations in the data arriving at that layer. Herein