跳到正文
返回论文列表
stat.ME提交于 已译

深度 M-回归的统计与隐私保证:基于随机梯度 Langevin 动力学

Statistical and Privacy Guarantees for Deep M-Regression via Stochastic Gradient Langevin Dynamics

Yeheng Ge · Jian Huang · Shuyuan Wu

中文摘要

研究通过随机梯度 Langevin 动力学(SGLD)训练的深度非参数回归,SGLD 是在优化过程中注入校准高斯噪声的随机梯度下降变体。针对深度 ReLU 网络采用 PAC-Bayes 分析,建立超额风险界,同时计入网络逼近误差、统计复杂度与 SGLD 轨迹累积的算法代价。在 Bernstein 条件与稳定 SGLD 机制下,超额风险界含一个系数与注入噪声方差成反比的随机项,而估计量在 Hölder 光滑条件下保持非参数回归的最优速率(至多对数因子)。该结果为适度 SGLD 噪声的泛化收益提供逼近误差感知、速率最优的统计解释。对逐样本裁剪 SGLD,进一步建立精确 f-DP 保证及相应渐近 μ-GDP 保证。给出 SGLD 训练 DNN 估计量的统计收敛速率与差分隐私保证,并在超额风险界中显式计入网络逼近误差与注入噪声影响。大量模拟与两个真实数据集分析考察有限样本预测性能与隐私-效用行为。

关键要点

  1. 01用 PAC-Bayes 分析给出深度 ReLU 网络 SGLD 训练的超额风险界,同时计入逼近误差、统计复杂度与算法代价。
  2. 02在 Bernstein 条件与稳定 SGLD 机制下,噪声方差越大随机项系数越小,估计量保持 Hölder 光滑下的最优非参数速率(至多对数因子)。
  3. 03对逐样本裁剪 SGLD 建立精确 f-DP 与渐近 μ-GDP 隐私保证。
  4. 04模拟实验与两个真实数据集验证有限样本预测性能及隐私-效用权衡。

解读

尚无解读。

原始英文摘要

arXiv:2611.07490v1 Announce Type: cross Abstract: In this article, we study deep nonparametric regression trained via stochastic gradient Langevin dynamics (SGLD), a variant of stochastic gradient descent that injects carefully calibrated Gaussian noise during optimization. Using a PAC-Bayesian analysis tailored to deep ReLU networks, we establish an excess-risk bound that simultaneously accounts for network approximation error, statistical complexity, and the algorithmic cost accumulated along the SGLD trajectory. Under a Bernstein condition and within a stable SGLD regime, the excess-risk bound contains a stochastic term whose coefficient is inversely proportional to the variance of injected noise, while the estimator retains the optimal rate for nonparametric regression under the H\"older smoothness condition up to logarithmic factors. This result provides an approximation-error-aware, rate-optimal statistical perspective on the widely observed generalization benefits of moderate SGLD noise. For per-sample clipped SGLD, we further establish an exact f-DP guarantee and the corresponding asymptotic $\mu$-GDP guarantee for the proposed estimator. We establish statistical convergence rates and differential privacy guarantees for DNN estimators trained using SGLD, while explicitly accounting for both network approximation error and the effect of injected noise in the excess-risk bound. Extensive simulation studies and analyses of two real datasets examine the finite-sample predictive performance and privacy-utility behavior of the proposed procedure.

同方向论文 · stat.ME

查看全部 →