We present a numerical method based on random projections with Gaussian kernels and physics-informed neural networks for the numerical solution of initial value problems (IVPs) of nonlinear stiff ordinary differential equations (ODEs) and index-1 differential algebraic equations (DAEs), which may also arise from spatial discretization of partial differential equations (PDEs). The internal weights are fixed to ones while the unknown weights between the hidden and output layer are computed with Newton’s iterations using the Moore–Penrose pseudo-inverse for low to medium scale and sparse QR decomposition with L 2 regularization for medium- to large-scale systems. Building on previous works on random projections, we also prove its approximation accuracy. To deal with stiffness and sharp gradients, we propose an adaptive step-size scheme and address a continuation method for providing good initial guesses for Newton iterations. The “optimal” bounds of the uniform distribution from which the values of the shape parameters of the Gaussian kernels are sampled and the number of basis functions are “parsimoniously” chosen based on bias-variance trade-off decomposition. To assess the performance of the scheme in terms of both numerical approximation accuracy and computational cost, we used eight benchmark problems (three index-1 DAEs problems, and five stiff ODEs problems including the Hindmarsh–Rose neuronal model of chaotic dynamics and the Allen–Cahn phase-field PDE). The efficiency of the scheme was compared against two stiff ODEs/DAEs solvers, namely, ode15s and ode23t solvers of the MATLAB ODE suite as well as against deep learning as implemented in the DeepXDE library for scientific machine learning and physics-informed learning for the solution of the Lotka–Volterra ODEs included in the demos of the library. A software/toolbox in Matlab (that we call RanDiffNet) with demos is also provided.
REFERENCES
1.
C. W.
Gear
, Numerical Initial Value Problems in Ordinary Differential Equations
(Prentice-Hall, Englewood Cliffs, NJ, 1971), pp. 1–253.2.
K. E.
Brenan
, S. L.
Campbell
, and L. R.
Petzold
, Numerical Solution of Initial-Value Problems in Differential-Algebraic Equations
(SIAM
, 1995
).3.
C.
Gear
, “An introduction to numerical methods for odes and daes,” in Real-Time Integration Methods for Mechanical System Simulation (Springer, 1990), pp. 115–126.4.
L. F.
Shampine
and M. W.
Reichelt
, “The MATLAB ODE suite
,” SIAM J. Scientific Comput.
18
, 1
–22
(1997
). 5.
K.
Krischer
, R.
Rico-Martinez
, I. G.
Kevrekidis
, H.
Rotermund
, G.
Ertl
, and J.
Hudson
, “Model identification of a spatiotemporally varying catalytic reaction
,” Aiche J.
39
, 89
–98
(1993
). 6.
S. F.
Masri
, A. G.
Chassiakos
, and T. K.
Caughey
, “Identification of nonlinear dynamic systems using neural networks
,” J. Appl. Mech.
60
, 123
–133
(1993
). 7.
T.
Chen
and H.
Chen
, “Universal approximation to nonlinear operators by neural networks with arbitrary activation functions and its application to dynamical systems
,” IEEE Trans. Neural Netw.
6
, 911
–917
(1995
).8.
A.
Taprantzis
, C.
Siettos
, and G.
Bafas
, “Fuzzy control of a fluidized bed dryer
,” Drying Technol.
15
, 511
–537
(1997
). 9.
R.
González-García
, R.
Rico-Martìnez
, and I. G.
Kevrekidis
, “Identification of distributed parameter systems: A neural net based approach
,” Comput. Chem. Eng.
22
, S965
–S968
(1998
).10.
C.
Siettos
, C.
Kiranoudis
, and G.
Bafas
, “Advanced control strategies for fluidized bed dryers
,” Drying Technol.
17
, 2271
–2291
(1999
). 11.
D.
Sagias
, E.
Sarafis
, C. I.
Siettos
, and G. V.
Bafas
, “Design of a model identification fuzzy adaptive controller and stability analysis of nonlinear processes
,” Fuzzy Sets Syst.
121
, 169
–179
(2001
). 12.
C. I.
Siettos
, G. V.
Bafas
, and A. G.
Boudouvis
, “Truncated chebyshev series approximation of fuzzy systems for control and nonlinear system identification
,” Fuzzy Sets Syst.
126
, 89
–104
(2002
). 13.
C. I.
Siettos
and G. V.
Bafas
, “Semiglobal stabilization of nonlinear systems using fuzzy control and singular perturbation methods
,” Fuzzy Sets and Systems
129
, 275
–294
(2002
). 14.
A.
Alexandridis
, C.
Siettos
, H.
Sarimveis
, A.
Boudouvis
, and G.
Bafas
, “Modelling of nonlinear process dynamics using Kohonen’s neural networks, fuzzy systems and chebyshev series
,” Comput. Chem. Eng.
26
, 479
–486
(2002
). 15.
H.
Lee
and I. S.
Kang
, “Neural algorithm for solving differential equations
,” J. Comput. Phys.
91
, 110
–131
(1990
). 16.
M.
Dissanayake
and N.
Phan-Thien
, “Neural-network-based approximations for solving partial differential equations
,” Commun. Numer. Meth. Eng.
10
, 195
–201
(1994
). 17.
A. J.
Meade
, Jr., and A. A.
Fernandez
, “The numerical solution of linear ordinary differential equations by feedforward neural networks
,” Math. Computer Modell.
19
, 1
–25
(1994
).18.
R.
Gerstberger
and P.
Rentrop
, “Feedforward neural nets as discretization schemes for ODEs and DAEs
,” J. Computat. Appl. Math.
82
, 117
–128
(1997
). 19.
I. E.
Lagaris
, A.
Likas
, and D. I.
Fotiadis
, “Artificial neural networks for solving ordinary and partial differential equations
,” IEEE Trans. Neural Netw.
9
, 987
–1000
(1998
). 20.
G. E.
Karniadakis
, I. G.
Kevrekidis
, L.
Lu
, P.
Perdikaris
, S.
Wang
, and L.
Yang
, “Physics-informed machine learning
,” Nat. Rev. Phys.
3
, 422
–440
(2021
). 21.
J.
Bongard
and H.
Lipson
, “Automated reverse engineering of nonlinear dynamical systems
,” Proc. Natl. Acad. Sci. U.S.A.
104
, 9943
–9948
(2007
). 22.
M.
Raissi
, P.
Perdikaris
, and G. E.
Karniadakis
, “Machine learning of linear differential equations using gaussian processes
,” J. Comput. Phys.
348
, 683
–693
(2017
). 23.
M.
Raissi
, P.
Perdikaris
, and G. E.
Karniadakis
, “Numerical Gaussian processes for time-dependent and nonlinear partial differential equations
,” SIAM J. Scientific Comput.
40
, A172
–A198
(2018
). 24.
M.
Raissi
, P.
Perdikaris
, and G. E.
Karniadakis
, “Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations
,” J. Comput. Phys.
378
, 686
–707
(2019
). 25.
S.
Rudy
, A.
Alla
, S. L.
Brunton
, and J. N.
Kutz
, “Data-driven identification of parametric partial differential equations
,” SIAM J. Appl. Dynam. Syst.
18
, 643
–660
(2019
). 26.
T.
Bertalan
, F.
Dietrich
, I.
Mezić
, and I. G.
Kevrekidis
, “On learning hamiltonian systems from data
,” Chaos
29
, 121107
(2019
). 27.
H.
Arbabi
, J. E.
Bunder
, G.
Samaey
, A. J.
Roberts
, and I. G.
Kevrekidis
, “Linking machine learning with multiscale numerics: Data-driven discovery of homogenized equations
,” JOM
72
, 4444
–4457
(2020
). 28.
S.
Lee
, M.
Kooshkbaghi
, K.
Spiliotis
, C. I.
Siettos
, and I. G.
Kevrekidis
, “Coarse-scale pdes from fine-scale observations via machine learning
,” Chaos
30
, 013141
(2020
). 29.
P. R.
Vlachas
, J.
Pathak
, B. R.
Hunt
, T. P.
Sapsis
, M.
Girvan
, E.
Ott
, and P.
Koumoutsakos
, “Backpropagation algorithms and reservoir computing in recurrent neural networks for the forecasting of complex spatiotemporal dynamics
,” Neural Netw.
126
, 191
–217
(2000
). 30.
W.
Chen
, Q.
Wang
, J. S.
Hesthaven
, and C.
Zhang
, “Physics-informed machine learning for reduced-order modeling of nonlinear problems
,” J. Comput. Phys.
446
, 110666
(2021
). 31.
Y.
Chen
, B.
Hosseini
, H.
Owhadi
, and A. M.
Stuart
, “Solving and learning nonlinear pdes with gaussian processes
,” J. Comput. Phys.
447
, 110668
(2021
). 32.
L.
Lu
, P.
Jin
, G.
Pang
, Z.
Zhang
, and G. E.
Karniadakis
, “Learning nonlinear operators via deeponet based on the universal approximation theorem of operators
,” Nat. Machine Intelligence
3
, 218
–229
(2021
). 33.
S.
Kim
, W.
Ji
, S.
Deng
, Y.
Ma
, and C.
Rackauckas
, “Stiff neural ordinary differential equations
,” Chaos
31
, 093122
(2021
). 34.
J.
Han
, A.
Jentzen
, and E.
Weinan
, “Solving high-dimensional partial differential equations using deep learning
,” Proc. Natl. Acad. Sci. U.S.A.
115
, 8505
–8510
(2018
). 35.
G.
Fabiani
, F.
Calabrò
, L.
Russo
, and C.
Siettos
, “Numerical solution and bifurcation analysis of nonlinear partial differential equations with extreme learning machines
,” J. Scientific Comput.
89
, 44
(2021
). 36.
F.
Calabrò
, G.
Fabiani
, and C.
Siettos
, “Extreme learning machine collocation for the numerical solution of elliptic PDEs with sharp gradients
,” Computer Meth. Appl. Mech. Eng.
387
, 114188
(2021
).37.
S.
Dong
and Z.
Li
, “Local extreme learning machines and domain decomposition for solving linear and nonlinear partial differential equations
,” Computer Meth. Appl. Mech. Eng.
387
, 114129
(2021
). 38.
S.
Dong
and Z.
Li
, “A modified batch intrinsic plasticity method for pre-training the random coefficients of extreme learning machines
,” J. Comput. Phys.
445
, 110585
(2021
). 39.
W.
Ji
, W.
Qiu
, Z.
Shi
, S.
Pan
, and S.
Deng
, “Stiff-pinn: Physics-informed neural network for stiff chemical kinetics
,” J. Phys. Chem. A
125
, 8098
–8106
(2021
). 40.
L.
Lu
, X.
Meng
, Z.
Mao
, and G. E.
Karniadakis
, “DeepXDE: A deep learning library for solving differential equations
,” SIAM Rev.
63
, 208
–228
(2021
). 41.
E.
Schiassi
, R.
Furfaro
, C.
Leake
, M.
De Florio
, H.
Johnston
, and D.
Mortari
, “Extreme theory of functional connections: A fast physics-informed neural network method for solving ordinary and partial differential equations
,” Neurocomputing
457
, 334
–356
(2021
). 42.
M.
De Florio
, E.
Schiassi
, and R.
Furfaro
, “Physics-informed neural networks and functional interpolation for stiff chemical kinetics
,” Chaos
32
, 063107
(2022
). 43.
Y.
Lu
, R.
Maulik
, T.
Gao
, F.
Dietrich
, I. G.
Kevrekidis
, and J.
Duan
, “Learning the temporal evolution of multivariate densities via normalizing flows
,” Chaos
32
, 033121
(2022
). 44.
S.
Dong
and J.
Yang
, “On computing the hyperparameter of extreme learning machines: Algorithm and application to computational pdes, and comparison with classical and high-order finite elements
,” J. Comput. Phys.
463
, 111290
(2022
). 45.
X.
Meng
, Z.
Li
, D.
Zhang
, and G. E.
Karniadakis
, “PPINN: Parareal physics-informed neural network for time-dependent PDEs
,” Computer Meth. Appl. Mech. Eng.
370
, 113250
(2020
). 46.
H.
Larochelle
, Y.
Bengio
, J.
Louradour
, and P.
Lamblin
, “Exploring strategies for training deep neural networks
,” J. Machine Learning Res.
10
, 1–40
(2009
).47.
S.
Wang
, Y.
Teng
, and P.
Perdikaris
, “Understanding and mitigating gradient flow pathologies in physics-informed neural networks
,” SIAM J. Scientific Comput.
43
, A3055
–A3081
(2021
). 48.
S.
Wang
, X.
Yu
, and P.
Perdikaris
, “When and why pinns fail to train: A neural tangent kernel perspective
,” J. Comput. Phys.
449
, 110768
(2022
). 49.
W. B.
Johnson
and J.
Lindenstrauss
, “Extensions of Lipschitz mappings into a Hilbert space
,” Contemp. Math.
26
, 189
–206
(1984
).50.
A.
Rahimi
and B.
Recht
, “Weighted sums of random kitchen sinks: Replacing minimization with randomization in learning,” in Advances in Neural Information Processing Systems 21 (NIPS 2008) (Citeseer, 2008), pp. 1313–1320.51.
A. N.
Gorban
, I. Y.
Tyukin
, D. V.
Prokhorov
, and K. I.
Sofeikov
, “Approximation with random bases: Pro et Contra
,” Inf. Sci.
364
, 129
–145
(2016
). 52.
Y.-H.
Pao
and Y.
Takefuji
, “Functional-link net computing: Theory, system architecture, and functionalities
,” Computer
25
, 76
–79
(1992
). 53.
H.
Jaeger
, “Adaptive nonlinear system identification with echo state networks
,” Adv. Neural Inform. Process. Syst.
15
, 609
–616
(2002
).54.
G.-B.
Huang
, Q.-Y.
Zhu
, and C.-K.
Siew
, “Extreme learning machine: Theory and applications
,” Neurocomputing
70
, 489
–501
(2006
). 55.
M.
Belkin
, D.
Hsu
, S.
Ma
, and S.
Mandal
, “Reconciling modern machine-learning practice and the classical bias–variance trade-off
,” Proc. Natl. Acad. Sci. U.S.A.
116
, 15849
–15854
(2019
). 56.
T. A.
Davis
, Direct Methods for Sparse Linear Systems
(SIAM, 2006).57.
H.
Robertson
, “The solution of a set of reaction rate equations
,” in Numerical Analysis: An Introduction
(Academic Press, London, 1966), pp. 178–182.58.
L. F.
Shampine
, M. W.
Reichelt
, and J. A.
Kierzenka
, “Solving index-1 daes in matlab and simulink
,” SIAM Rev.
41
, 538
–552
(1999
). 59.
J. L.
Hindmarsh
and R.
Rose
, “A model of neuronal bursting using three coupled first order differential equations
,” Proc. R. Soc. London Ser. B
221
, 87
–102
(1984
).60.
B. P.
Belousov
, “A periodic reaction and its mechanism
,” in Oscillations and Traveling Waves in Chemical Systems
, edited by R. J. Field and M. Burger (Wiley, New York, 1985).61.
A. M.
Zhabotinsky
, “Periodic course of the oxidation of malonic acid in a solution (Studies on the kinetics of Beolusov's reaction)
,” Biofizika
9
, 306
–311
(1964
).62.
S. M.
Allen
and J. W.
Cahn
, “A microscopic theory for antiphase boundary motion and its application to antiphase domain coarsening
,” Acta Metall.
27
, 1085
–1095
(1979
). 63.
64.
G.-B.
Huang
, L.
Chen
, and C. K.
Siew
, “Universal approximation using incremental constructive feedforward networks with random hidden nodes
,” IEEE Trans. Neural Netw.
17
, 879
–892
(2006
). 65.
G.
Söderlind
, “Automatic control and adaptive time-stepping
,” Numer. Algorithms
31
, 281
–310
(2002
).66.
L. F.
Shampine
and C. W.
Gear
, “A user’s view of solving stiff ordinary differential equations
,” SIAM Rev.
21
, 1
–17
(1979
). 67.
A. G.
Baydin
, B. A.
Pearlmutter
, A. A.
Radul
, and J. M.
Siskind
, “Automatic differentiation in machine learning: A survey
,” J. Machine Learn. Res.
18
, 1–43
(2018
).68.
B.
Igelnik
and Y.-H.
Pao
, “Stochastic choice of basis functions in adaptive function approximation and the functional-link net
,” IEEE Trans. Neural Netw.
6
, 1320
–1329
(1995
). 69.
D.
Husmeier
, “Random vector functional link (RVFL) networks,” in Neural Networks for Conditional Probability Estimation (Springer, 1999), pp. 87–97.70.
D. J.
Gauthier
, E.
Bollt
, A.
Griffith
, and W. A.
Barbosa
, “Next generation reservoir computing
,” Nat. Commun.
12
, 1
–8
(2021
). 71.
L. A.
Thiede
and U.
Parlitz
, “Gradient based hyperparameter optimization in echo state networks
,” Neural Netw.
115
, 23
–29
(2019
). 72.
G.-B.
Huang
, “An insight into extreme learning machines: Random neurons, random features and kernels
,” Cognit. Computat.
6
, 376
–390
(2014
). 73.
B.
Ghojogh
, A.
Ghodsi
, F.
Karray
, and M.
Crowley
, “Johnson-lindenstrauss lemma, linear and nonlinear random projections, random fourier features, and random kitchen sinks: Tutorial and survey,” arXiv:2108.04172 (2021).74.
R.
Giryes
, G.
Sapiro
, and A. M.
Bronstein
, “Deep neural networks with random Gaussian weights: A universal classification strategy?
,” IEEE Trans. Signal Process.
64
, 3444
–3457
(2016
). 75.
76.
A. R.
Barron
, “Universal approximation bounds for superpositions of a sigmoidal function
,” IEEE Trans. Inf. Theory
39
, 930
–945
(1993
). 77.
A.
Rahimi
and B.
Recht
, “Uniform approximation of functions with random bases,” in 2008 46th Annual Allerton Conference on Communication, Control, and Computing (IEEE, 2008), pp. 555–561.78.
P.
Collins
, and O. U. K. M. Inst.
, Differential and Integral Equations: Part II
(University of Oxford Mathematical Institute
, 1988
).79.
N.
De Villiers
and D.
Glasser
, “A continuation method for nonlinear regression
,” SIAM J. Numer. Anal.
18
, 1139
–1154
(1981
). 80.
T. A.
Davis
, “Algorithm 915, suitesparseqr: Multifrontal multithreaded rank-revealing sparse QR factorization
,” ACM Trans. Math. Software (TOMS)
38
, 1
–22
(2011
).81.
I.
Gladwell
, L.
Shampine
, and R.
Brankin
, “Automatic selection of the initial step size for an ode solver
,” J. Computat. Appl. Math.
18
, 175
–192
(1987
). 82.
E.
Hairer
, S. P.
Norsett
, and G.
Wanner
, Solving Ordinary, Differential Equations I, Nonstiff Problems, with 135 Figures
, 2nd ed.
(Springer-Verlag
, 2000
), Vol. 1.83.
W. H.
Enright
, K. R.
Jackson
, S. P.
Nørsett
, and P. G.
Thomsen
, “Interpolants for Runge-Kutta formulas
,” ACM Trans. Math. Software (TOMS)
12
, 193
–218
(1986
). 84.
A.
Prothero
and A.
Robinson
, “On the stability and accuracy of one-step methods for solving stiff systems of ordinary differential equations
,” Math. Comput.
28
, 145
–162
(1974
). 85.
L.
Métivier
and P.
Montarnal
, “Strategies for solving index one dae with non-negative constraints: Application to liquid–liquid extraction
,” J. Comput. Phys.
231
, 2945
–2962
(2012
).86.
R.
Belusov
, “Periodicheski deistvuyushchaya reaktsia i ee mekhanism \{Periodically acting reaction and its mechanism\},” in Sbornik referatov po radiotsionnoi meditsine (Collection of Abstracts on Radiation Medicine
) (Medgiz, Moscow, 1958), pp. 145–147.87.
V.
Shulyk
, O.
Klymenko
, and I.
Svir
, “Numerical solution of stiff odes describing complex homogeneous chemical processes
,” J. Math. Chem.
43
, 252–264
(2008
). 88.
P. N.
Brown
, A. C.
Hindmarsh
, and L. R.
Petzold
, “Using Krylov methods in the solution of large-scale differential-algebraic systems
,” SIAM J. Scientific Comput.
15
, 1467
–1488
(1994
). 89.
90.
L. R.
Petzold
, “Description of dassl: A differential/algebraic system solver,” Tech. Rep. (Sandia National Labs., Livermore, CA (USA), 1982).91.
M.
Hadjinicolaou
and D. A.
Goussis
, “Asymptotic solution of stiff pdes with the CSP method: The reaction diffusion equation
,” SIAM J. Scientific Comput.
20
, 781
–810
(1998
). 92.
D. T.
Doncevic
, A.
Mitsos
, Y.
Guo
, Q.
Li
, F.
Dietrich
, M.
Dahmen
, and I. G.
Kevrekidis
, “A recursively recurrent neural network (R2N2) architecture for learning iterative algorithms,” arXiv:2211.12386 (2022).93.
Y.
Guo
, F.
Dietrich
, T.
Bertalan
, D. T.
Doncevic
, M.
Dahmen
, I. G.
Kevrekidis
, and Q.
Li
, “Personalized algorithm generation: A case study in learning ode integrators
,” SIAM J. Scientific Comput.
44
, A1911
–A1933
(2022
). 94.
W. F.
Schmidt
, M. A.
Kraaijveld
, and R. P. W.
Duin
, “Feedforward neural networks with random weights
,” Proceedings, 11th IAPR International Conference on Pattern Recognition. Vol.II. Conference B: Pattern Recognition Methodology and Systems
, The Hague, Netherlands, 30 Aug.–3 Sep. 1992 (IEEE, 1992).95.
D. S.
Broomhead
and D.
Lowe
“Multivariable functional interpolation and adaptive networks
,” Complex Systems
2
, 321–355 (1988).© 2023 Author(s). Published under an exclusive license by AIP Publishing.
2023
Author(s)
You do not currently have access to this content.