Neural networks have the ability to serve as universal function approximators, but they are not interpretable and do not generalize well outside of their training region. Both of these issues are problematic when trying to apply standard neural ordinary differential equations (ODEs) to dynamical systems. We introduce the polynomial neural ODE, which is a deep polynomial neural network inside of the neural ODE framework. We demonstrate the capability of polynomial neural ODEs to predict outside of the training region, as well as to perform direct symbolic regression without using additional tools such as SINDy.
REFERENCES
1.
S. L.
Brunton
, J. L.
Proctor
, and J. N.
Kutz
, “Discovering governing equations from data by sparse identification of nonlinear dynamical systems
,” Proc. Natl. Acad. Sci. U.S.A.
113
, 3932
–3937
(2016
). 2.
E. P.
Alves
and F.
Fiuza
, “Robust data-driven discovery of reduced plasma physics models from fully kinetic simulations,” in APS Division of Plasma Physics Meeting Abstracts, APS Meeting Abstracts (APS, 2020), Vol. 2020, pp. GO10.006.3.
M.
Sorokina
, S.
Sygletos
, and S.
Turitsyn
, “Sparse identification for nonlinear optical communication systems: Sino method
,” Opt. Express
24
, 30433
–30443
(2016
). 4.
N. M.
Mangan
, S. L.
Brunton
, J. L.
Proctor
, and J. N.
Kutz
, “Inferring biological networks by sparse identification of nonlinear dynamics
,” IEEE Trans. Mol., Biol. Multi-Scale Commun.
2
, 52
–63
(2016
). 5.
M.
Hoffmann
, C.
Fröhner
, and F.
Noé
, “Reactive SINDy: Discovering governing reactions from concentration data
,” J. Chem. Phys.
150
, 025101
(2019
). 6.
S.
Rudy
, S.
Brunton
, J.
Proctor
, and J.
Kutz
, “Data-driven discovery of partial differential equations
,” Sci. Adv.
3
, e1602614
(2016
). 7.
L. M.
Mayr
and D.
Bojanic
, “Novel trends in high-throughput screening
,” Curr. Opin. Pharmacol.
9
, 580
–588
(2009
). 8.
P.
Szymański
, M.
Markowicz
, and E.
Mikiciuk-Olasik
, “Adaptation of high-throughput screening in drug discovery—Toxicological screening tests
,” Int. J. Mol. Sci.
13
, 427
–452
(2011
). 9.
T.
Kalsoom
, N.
Ramzan
, S.
Ahmed
, and M.
Ur Rehman
, “Advances in sensor technologies in the era of smart factory and industry 4.0
,” Sensors
20
, 6783
(2020
). 10.
G.
Balsamo
, A.
Agusti-Panareda
, C.
Albergel
, G.
Arduini
, A.
Beljaars
, J.
Bidlot
, N.
Bousserez
, S.
Boussetta
, A.
Brown
, R.
Buizza
, C.
Buontempo
, F.
Chevallier
, M.
Choulga
, H.
Cloke
, M.
Cronin
, M.
Dahoui
, P.
Rosnay
, P.
Dirmeyer
, E.
Dutra
, and X.
Zeng
, “Satellite and in situ observations for advancing global earth surface modelling: A review
,” Remote Sens.
10
, 2038
(2018
). 11.
G.
Calzolari
and W.
Liu
, “Deep learning to replace, improve, or aid CFD analysis in built environment applications: A review
,” Build. Environ.
206
, 108315
(2021
). 12.
D. A.
Randall
, R. A.
Wood
, S.
Bony
, R.
Colman
, T.
Fichefet
, J.
Fyfe
, V.
Kattsov
, A.
Pitman
, J.
Shukla
, J.
Srinivasan
, and R. J.
Stouffer
, “Climate models and their evaluation,” in Climate Change 2007: The Physical Science Basis. Contribution of Working Group I to the Fourth Assessment Report of the IPCC (FAR) (Cambridge University Press, 2007), pp. 589–662.13.
N.
Maher
, S.
Milinski
, and R.
Ludwig
, “Large ensemble climate model simulations: Introduction, overview, and future prospects for utilising multiple types of large ensemble
,” Earth Syst. Dyn.
12
, 401
–418
(2021
). 14.
R. T.
Chen
, Y.
Rubanova
, J.
Bettencourt
, and D. K.
Duvenaud
, “Neural ordinary differential equations
,” Adv. Neural Inf. Process. Syst.
31
, 6571
–6583
(2018
).15.
Y.
Rubanova
, R. T. Q.
Chen
, and D. K.
Duvenaud
, “Latent ordinary differential equations for irregularly-sampled time series,” in Advances in Neural Information Processing Systems, edited by H. Wallach, H. Larochelle, A. Beygelzimer, F. d’Alché-Buc, E. Fox, and R. Garnett (Curran Associates, Inc., 2019), Vol. 32.16.
R.
Dandekar
, V.
Dixit
, M.
Tarek
, A.
Garcia-Valadez
, and C.
Rackauckas
, “Bayesian neural ordinary differential equations,” CoRR abs/2012.07244 (2020).17.
X.
Li
, T.-K. L.
Wong
, R. T. Q.
Chen
, and D.
Duvenaud
, “Scalable gradients for stochastic differential equations,” in Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, Proceedings of Machine Learning Research, edited by S. Chiappa and R. Calandra (PMLR, 2020), Vol. 108, pp. 3870–3882.18.
C.
Rackauckas
, Y.
Ma
, J.
Martensen
, C.
Warner
, K.
Zubov
, R.
Supekar
, D.
Skinner
, and A. J.
Ramadhan
, “Universal differential equations for scientific machine learning,” CoRR abs/2001.04385 (2020).19.
K.
Champion
, S.
Brunton
, and J.
Kutz
, “Discovery of nonlinear multiscale systems: Sampling strategies and embeddings
,” SIAM J. Appl. Dyn. Syst.
18
, 312
–333
(2019
). 20.
K.
Lee
, N.
Trask
, and P.
Stinis
, “Structure-preserving sparse identification of nonlinear dynamics for data-driven modeling,” arXiv:2109.05364 (2021).21.
I.
Goodfellow
, Y.
Bengio
, and A.
Courville
, Deep Learning
(MIT Press
, 2016
), see http://www.deeplearningbook.org.22.
G.
Sanguinetti
and V.
Huynh-Thu
, “Gene regulatory networks: Methods and protocols,” in Methods in Molecular Biology (Springer New York, 2018).23.
J.
Gutkind
, “Signaling networks and cell cycle control: The molecular basis of cancer and other diseases,” in Cancer Drug Discovery and Development (Humana Press, 2000).24.
J.
Royle
and R.
Dorazio
, Hierarchical Modeling and Inference in Ecology: The Analysis of Data from Populations, Metapopulations and Communities
(Elsevier Science
, 2008
).25.
H.
Singh
and J.
Dhar
, Mathematical Population Dynamics and Epidemiology in Temporal and Spatio-Temporal Domains
(Apple Academic Press
, 2018
).26.
G.
Brasseur
and D.
Jacob
, Modeling of Atmospheric Chemistry
(Cambridge University Press
, 2017
).27.
G. G.
Chrysos
, S.
Moschoglou
, G.
Bouritsas
, J.
Deng
, Y.
Panagakis
, and S.
Zafeiriou
, “Deep polynomial neural networks
,” IEEE Trans. Pattern Anal. Mach. Intell.
44
, 4021
–4034
(2022
). 28.
J.
Distefano
, Dynamic Systems Biology Modeling and Simulation
(Elsevier Science
, 2015
).29.
A.
Andoni
, R.
Panigrahy
, G.
Valiant
, and L.
Zhang
, “Learning polynomials with neural networks,” in Proceedings of the 31st International Conference on Machine Learning, Proceedings of Machine Learning Research, edited by E. P. Xing and T. Jebara (PMLR, Bejing, 2014), Vol. 32, pp. 1908–1916.30.
F.
Fan
, J.
Xiong
, and G.
Wang
, “Universal approximation with quadratic deep networks
,” Neural Netw.
124
, 383
–392
(2020
). 31.
S.
Du
and J.
Lee
, “On the power of over-parametrization in neural networks with quadratic activation,” in International Conference on Machine Learning (PMLR, 2018), pp. 1329–1338.32.
S.
Liang
and R.
Srikant
, “Why deep neural networks for function approximation?,” arXiv:1610.04161 (2016).33.
R.
Horn
, R.
Horn
, and C.
Johnson
, Topics in Matrix Analysis
(Cambridge University Press
, 1994
).34.
A.
Meurer
, C. P.
Smith
, M.
Paprocki
, O.
Čertík
, S. B.
Kirpichev
, M.
Rocklin
, A.
Kumar
, S.
Ivanov
, J. K.
Moore
, S.
Singh
, T.
Rathnayake
, S.
Vig
, B. E.
Granger
, R. P.
Muller
, F.
Bonazzi
, H.
Gupta
, S.
Vats
, F.
Johansson
, F.
Pedregosa
, M. J.
Curry
, A. R.
Terrel
, V.
Roučka
, A.
Saboo
, I.
Fernando
, S.
Kulal
, R.
Cimrman
, and A.
Scopatz
, “Sympy: Symbolic computing in python
,” PeerJ Comput. Sci.
3
, e103
(2017
). 35.
E.
Fehlberg
, Classical Fifth-, Sixth-, Seventh-, and Eighth-Order Runge-Kutta Formulas with Stepsize Control
(National Aeronautics and Space Administration
, 1968
).36.
H.
Wilbraham
, “On a certain periodic function
,” Cambridge Dublin Math. J.
3
, 198
–201
(1848
).37.
A.
Jerri
, “The gibbs phenomenon in fourier analysis, splines and wavelet approximations,” in Mathematics and Its Applications (Springer US, 2013).38.
39.
V.
Volterra
, Variazioni E Fluttuazioni Del Numero D’individui in Specie Animali Conviventi
(Società anonima tipografica “Leonardo da Vinci,”
1926
).40.
P.
Virtanen
, R.
Gommers
, T. E.
Oliphant
, M.
Haberland
, T.
Reddy
, D.
Cournapeau
, E.
Burovski
, P.
Peterson
, W.
Weckesser
, J.
Bright
, S. J.
van der Walt
, M.
Brett
, J.
Wilson
, K. J.
Millman
, N.
Mayorov
, A. R. J.
Nelson
, E.
Jones
, R.
Kern
, E.
Larson
, C. J.
Carey
, İ.
Polat
, Y.
Feng
, E. W.
Moore
, J.
VanderPlas
, D.
Laxalde
, J.
Perktold
, R.
Cimrman
, I.
Henriksen
, E. A.
Quintero
, C. R.
Harris
, A. M.
Archibald
, A. H.
Ribeiro
, F.
Pedregosa
, and P.
van Mulbregt
, and SciPy 1.0 Contributors
, “SciPy 1.0: Fundamental algorithms for scientific computing in Python
,” Nat. Methods
17
, 261
–272
(2020
). 41.
R. T. Q.
Chen
, “torchdiffeq” (2018), see https://github.com/rtqichen/torchdiffeq.42.
J.
Dormand
and P.
Prince
, “A family of embedded Runge-Kutta formulae
,” J. Comput. Appl. Math.
6
, 19
–26
(1980
). 43.
E.
Roesch
, C.
Rackauckas
, and M.
Stumpf
, “Collocation based training of neural ordinary differential equations
,” Stat. Appl. Genet. Mol. Biol.
20
, 37
–49
(2021
). 44.
B.
de Silva
, K.
Champion
, M.
Quade
, J.-C.
Loiseau
, J.
Kutz
, and S.
Brunton
, “Pysindy: A python package for the sparse identification of nonlinear dynamical systems from data
,” J. Open Source Softw.
5
, 2104
(2020
). 45.
A. A.
Kaptanoglu
, B. M.
de Silva
, U.
Fasel
, K.
Kaheman
, A. J.
Goldschmidt
, J.
Callaham
, C. B.
Delahunt
, Z. G.
Nicolaou
, K.
Champion
, J.-C.
Loiseau
, J. N.
Kutz
, and S. L.
Brunton
, “Pysindy: A comprehensive python package for robust sparse system identification
,” J. Open Source Softw.
7
, 3994
(2022
). 46.
P.
Zheng
, T.
Askham
, S. L.
Brunton
, J. N.
Kutz
, and A. Y.
Aravkin
, “A unified framework for sparse relaxed regularized regression: SR3
,” IEEE Access
7
, 1404
–1423
(2019
). 47.
N. B.
Janson
, “Non-linear dynamics of biological systems
,” Contemp. Phys.
53
, 137
–168
(2012
). 48.
D.
Karnopp
, D. L.
Margolis
, and R. C.
Rosenberg
, System Dynamics
(Wiley New York
, 1990
).49.
B.
van der Pol
, “On relaxation-oscillations
,” Lond., Edinb., Dublin Philos. Mag. J. Sci.
2
, 978
–992
(1926
). 50.
R.
FitzHugh
, “Impulses and physiological states in theoretical models of nerve membrane
,” Biophys. J.
1
, 445
–466
(1961
). 51.
J.
Nagumo
, S.
Arimoto
, and S.
Yoshizawa
, “An active pulse transmission line simulating nerve axon
,” Proc. IRE
50
, 2061
–2070
(1962
). 52.
J. H. E.
Cartwright
, V. M.
Eguíluz
, E.
Hernández-García
, and O.
Piro
, “Dynamics of elastic excitable media
,” Int. J. Bifurcat. Chaos
09
, 2197
–2202
(1999
). 53.
J. C.
Lucero
and J.
Schoentgen
, “Modeling vocal fold asymmetries with coupled van der Pol oscillators
,” Proc. Meet. Acoust.
19
, 060165
(2013
). 54.
L. F.
Shampine
and C. W.
Gear
, “A user’s view of solving stiff ordinary differential equations
,” SIAM Rev.
21
, 1
–17
(1979
). 55.
S.
Kim
, W.
Ji
, S.
Deng
, Y.
Ma
, and C.
Rackauckas
, “Stiff neural ordinary differential equations
,” Chaos
31
, 093122
(2021
). © 2023 Author(s). Published under an exclusive license by AIP Publishing.
2023
Author(s)
You do not currently have access to this content.