We start from the contractive functional equation proposed in [4], where it was shown that the polynomial solution of functional equation can be used to initialize a Neural Network structure, with a controlled accuracy. We propose a novel algorithm, where the functional equation is solved with a converging iterative algorithm which can be realized as a Machine Learning training method iteratively with respect to the number of layers. The proof of convergence is performed with respect to the norm. Numerical tests illustrate the theory and show that stochastic gradient descent methods can be used with good accuracy for this problem.
Accepté le :
Accepté après révision le :
Publié le :
Després, Bruno 1
CC-BY 4.0
@article{CRMATH_2023__361_G6_1029_0,
author = {Despr\'es, Bruno},
title = {A convergent {Deep} {Learning} algorithm for approximation of polynomials},
journal = {Comptes Rendus. Math\'ematique},
pages = {1029--1040},
year = {2023},
publisher = {Acad\'emie des sciences, Paris},
volume = {361},
number = {G6},
doi = {10.5802/crmath.462},
language = {en},
url = {https://www.numdam.org/articles/10.5802/crmath.462/}
}
TY - JOUR AU - Després, Bruno TI - A convergent Deep Learning algorithm for approximation of polynomials JO - Comptes Rendus. Mathématique PY - 2023 SP - 1029 EP - 1040 VL - 361 IS - G6 PB - Académie des sciences, Paris UR - https://www.numdam.org/articles/10.5802/crmath.462/ DO - 10.5802/crmath.462 LA - en ID - CRMATH_2023__361_G6_1029_0 ER -
%0 Journal Article %A Després, Bruno %T A convergent Deep Learning algorithm for approximation of polynomials %J Comptes Rendus. Mathématique %D 2023 %P 1029-1040 %V 361 %N G6 %I Académie des sciences, Paris %U https://www.numdam.org/articles/10.5802/crmath.462/ %R 10.5802/crmath.462 %G en %F CRMATH_2023__361_G6_1029_0
Després, Bruno. A convergent Deep Learning algorithm for approximation of polynomials. Comptes Rendus. Mathématique, Tome 361 (2023) no. G6, pp. 1029-1040. doi: 10.5802/crmath.462
[1] Deep Learning with Python, Manning Shelter Island, 2018
[2] Nonlinear Approximation and (Deep) ReLU Networks, Computing, Volume 55/1 (2022), pp. 127-172 | Zbl
[3] Neural Networks and Numerical Analysis, Walter de Gruyter, 2022 | DOI
[4] A functional equation with polynomial solutions and application to Neural Networks, C. R. Math. Acad. Sci. Paris, Volume 358 (2020) no. 9-10, pp. 1059-1072 | Numdam | MR | Zbl
[5] Nonlinear Approximation by Deep ReLU Neural Networks (2019) (https://devore2019.sciencesconf.org/resource/page/id/1)
[6] Constructive Approximation, Grundlehren der Mathematischen Wissenschaften, 303, Springer, 1993 | DOI
[7] Deep Learning, MIT Press, 2016 (http://www.deeplearningbook.org)
[8] The Takagi Function and Its Generalization, Japan J. Appl. Math., Volume 1 (1984) no. 1, pp. 83-199 | MR | Zbl
[9] ReLU Deep Neural Networks and Linear Finite Elements, J. Comput. Math., Volume 38 (2020) no. 3, pp. 502-527 | MR | Zbl
[10] Convex analysis and minimization algorithms. I, 305, Springer, 1993
[11] Quand la machine apprend, Odile Jacob, 2019
[12] Better approximations of high dimensional smooth functions by deep neural networks with rectified power units, Commun. Comput. Phys., Volume 27 (2020) no. 2, pp. 379-411 | Zbl | MR
[13] Deep ReLU networks and high-order finite element methods, Anal. Appl., Singap., Volume 18 (2020) no. 5, pp. 715-770 | DOI | MR | Zbl
[14] The convergence of the Stochastic Gradient Descent (SGD) : a self-contained proof (2021) | arXiv
[15] Representation formulas and pointwise properties for Barron functions (2021) | arXiv
[16] Error bounds for approximations with deep ReLU networks, Neural Netw., Volume 94 (2017), pp. 103-114 | Zbl | DOI
Cité par Sources :
