分解神经网络中的 SGD 动力学:教师诱导尖峰与方差膨胀
Decomposing SGD Dynamics in Neural Networks: Teacher-Induced Spikes and Variance Inflation
Samet Demir · Zafer Dogan
中文摘要
研究两层神经网络在多指标教师模型上用随机梯度下降(SGD)训练时,与教师子空间正交的第一层参数如何影响泛化。将第一层权重分解为与教师对齐的信号分量和正交补,并在比例极限下分析其与第二层的联合 SGD 动力学。结果表明,尽管 SGD 对补的更新初始是各向同性的,训练动力学会在补的 Gram 矩阵中诱导出依赖教师的谱尖峰。这种涌现的各向异性提供了表示方差被放大、泛化退化的具体机制。为隔离该效应,引入 Decomposed Dynamics,一个解析可处理的替代模型,其渐近测试误差动力学与 SGD 一致。在 ReLU 学生的模拟中,冻结正交补的方向相比训练它们能提升泛化。最后,信号分量快速与第二层权重对齐,揭示即使在此浅层设定下也存在强跨层耦合。这些结果为教师子空间之外的特征学习如何损害泛化提供了动力学解释。
关键要点
- 01研究两层网络 SGD 训练中正交于教师子空间的第一层参数对泛化的影响
- 02发现 SGD 在补 Gram 矩阵中诱导教师依赖的谱尖峰,导致表示方差放大与泛化退化
- 03提出 Decomposed Dynamics 替代模型,其渐近测试误差动力学与 SGD 一致
- 04ReLU 学生模拟中冻结正交补方向比训练它们泛化更好
- 05信号分量快速与第二层权重对齐,显示浅层网络中的强跨层耦合
解读
尚无解读。
原始英文摘要
arXiv:2611.24908v1 Announce Type: new Abstract: We study two-layer neural networks trained by stochastic gradient descent (SGD) on a multi-index teacher model, focusing on how first-layer parameters orthogonal to the teacher subspace affect generalization. We decompose the first-layer weights into a signal component aligned with the teacher and an orthogonal complement, and analyze their joint SGD dynamics together with the second layer in the proportional limit. We show that although SGD updates to the complement are initially isotropic, the training dynamics induce teacher-dependent spectral spikes in the complement Gram matrix. This emergent anisotropy provides a concrete mechanism by which representation variance is amplified, and generalization degrades. To isolate this effect, we introduce Decomposed Dynamics, an analytically tractable surrogate whose asymptotic test-error dynamics coincide with those of SGD. In simulations with a ReLU student, freezing the directions of the orthogonal complement improves generalization relative to training them. Finally, we show that the signal component rapidly aligns with the second-layer weights, revealing strong cross-layer coupling even in this shallow setting. Together, our results provide a dynamical explanation for how feature learning outside the teacher subspace harms generalization.