This paper deals with the distribution of singular values of the input–output Jacobian of deep untrained neural networks in the limit of their infinite width. The Jacobian is the product of random matrices where the independent weight matrices alternate with diagonal matrices whose entries depend on the corresponding column of the nearest neighbor weight matrix. The problem has been considered in the several recent studies of the field for the Gaussian weights and biases and also for the weights that are Haar distributed orthogonal matrices and Gaussian biases. Based on a free probability argument, it was claimed in those papers that, in the limit of infinite width (matrix size), the singular value distribution of the Jacobian coincides with that of the analog of the Jacobian with special random but weight independent diagonal matrices, the case well known in random matrix theory. In this paper, we justify the claim for random Haar distributed weight matrices and Gaussian biases. This, in particular, justifies the validity of the mean field approximation in the infinite width limit for the deep untrained neural networks and extends the macroscopic universality of random matrix theory to this new class of random matrices.
Skip Nav Destination
Article navigation
June 2022
Research Article|
June 29 2022
Eigenvalue distribution of large random matrices arising in deep neural networks: Orthogonal case
Special Collection:
Special collection in honor of Freeman Dyson
L. Pastur
L. Pastur
a)
(Conceptualization, Writing – original draft, Writing – review & editing)
B. Verkin Institute for Low Temperature Physics and Engineering
, Kharkiv, Ukraine
a)Author to whom correspondence should be addressed: [email protected]
Search for other works by this author on:
L. Pastur
a)
B. Verkin Institute for Low Temperature Physics and Engineering
, Kharkiv, Ukraine
a)Author to whom correspondence should be addressed: [email protected]
Note: This paper is part of the special collection in Honor of Freeman Dyson.
J. Math. Phys. 63, 063505 (2022)
Article history
Received:
January 13 2022
Accepted:
May 30 2022
Citation
L. Pastur; Eigenvalue distribution of large random matrices arising in deep neural networks: Orthogonal case. J. Math. Phys. 1 June 2022; 63 (6): 063505. https://doi.org/10.1063/5.0085204
Download citation file:
Pay-Per-View Access
$40.00
Sign In
You could not be signed in. Please check your credentials and make sure you have an active account and try again.
Citing articles via
Well-posedness and decay structure of a quantum hydrodynamics system with Bohm potential and linear viscosity
Ramón G. Plaza, Delyan Zhelyazov
A sufficient criterion for divisibility of quantum channels
Frederik vom Ende
Derivation of the Maxwell–Schrödinger equations: A note on the infrared sector of the radiation field
Marco Falconi, Nikolai Leopold
Related Content
Composite parameterization and Haar measure for all unitary and special unitary groups
J. Math. Phys. (January 2012)
Tensor models and 3-ary algebras
J. Math. Phys. (October 2011)
Construction of n-Lie algebras and n-ary Hom-Nambu-Lie algebras
J. Math. Phys. (December 2011)
Alcoholism detection in magnetic resonance imaging by Haar wavelet transform and back propagation neural network
AIP Conf. Proc. (April 2018)
The design and research of anti-color-noise chaos M-ary communication system
AIP Advances (March 2016)