跳到正文
返回论文列表
stat.ML提交于 已译

DL101 神经网络模型的表示能力

DL101 The Representation Capability of Neural Network Models

Fernando Berzal

中文摘要

通用逼近定理(Cybenko、Funahashi、Hornik & Leshno)为神经网络足以建模复杂非线性现象提供了数学依据。浅层网络可以逼近紧集上的任意连续函数。Leshno 定理阐明,非多项式激活函数是实现通用性的必要条件,这解释了 ReLU、logistic、双曲正切等常见激活函数为何有效(而纯多项式激活函数无效)。然而,逼近定理并不提供效率、训练或泛化方面的保证:它们既未说明达到给定精度需要多少神经元,也未说明梯度下降能否找到正确参数,更未说明在有限数据上训练的网络能否良好泛化。Barron 针对特定函数类给出了定量误差界(具有有界傅里叶范数的函数的逼近速率),而 Montúfar 首次证明表达能力随深度呈指数增长(以线性区域数量衡量)。在相同参数数量下,深层网络严格比浅层网络更具表达能力。深度从根本上比宽度更强大,这为使用深度学习模型提供了严格的数学解释。

关键要点

  1. 01通用逼近定理证明浅层网络可逼近紧集上任意连续函数,但仅保证存在性。
  2. 02Leshno 定理指出非多项式激活函数是通用性的必要条件,解释了 ReLU 等激活函数为何有效。
  3. 03逼近定理不提供效率、训练收敛或泛化保证,无法说明所需神经元数量或梯度下降能否找到正确参数。
  4. 04Barron 给出有界傅里叶范数函数类的定量逼近速率,Montúfar 证明表达能力随深度指数增长。
  5. 05相同参数下深层网络严格比浅层网络更具表达能力,深度从根本上强于宽度。

解读

尚无解读。

原始英文摘要

arXiv:2611.24980v1 Announce Type: new Abstract: Universal approximation theorems (Cybenko, Funahashi, Hornik & Leshno) provide the mathematical justification for why neural networks are powerful enough to model complex, nonlinear phenomena. Shallow networks can approximate any continuous function on compact sets. Leshno's theorem clarified that non-polynomial activations are necessary for universality, which explains why ReLU, logistic, hyperbolic tangent, and other common activations work (but pure polynomials do not). However, approximation theorems provide no efficiency, training, nor generalization guarantees. They do not indicate how many neurons are needed for a given accuracy, whether gradient descent will find the right parameters, or whether networks trained on finite data will generalize well. Barron provided quantitative error bounds for particular function classes (approximation rates for functions with bounded Fourier norm), whereas Mont\'ufar was the first to show that expressivity grows exponentially with depth (in terms of the number of linear regions). Deep networks are strictly more expressive than shallow ones with the same number of parameters. Depth is fundamentally more powerful than width, providing a rigorous mathematical explanation for the use of deep learning models.

同方向论文 · stat.ML

查看全部 →