Using an OpenMP Application Programming Interface, the resolution-of-the-identity second-order Møller–Plesset perturbation (RI-MP2) method has been off-loaded onto graphical processing units (GPUs), both as a standalone method in the GAMESS electronic structure program and as an electron correlation energy component in the effective fragment molecular orbital (EFMO) framework. First, a new scheme has been proposed to maximize data digestion on GPUs that subsequently linearizes data transfer from central processing units (CPUs) to GPUs. Second, the GAMESS Fortran code has been interfaced with GPU numerical libraries (e.g., NVIDIA cuBLAS and cuSOLVER) for efficient matrix operations (e.g., matrix multiplication, matrix decomposition, and matrix inversion). The standalone GPU RI-MP2 code shows an increasing speedup of up to 7.5× using one NVIDIA V100 GPU with one IBM 42-core P9 CPU for calculations on fullerenes of increasing size from 40 to 260 carbon atoms using the 6-31G(d)/cc-pVDZ-RI basis sets. A single Summit node with six V100s can compute the RI-MP2 correlation energy of a cluster of 175 water molecules using the correlation consistent basis sets cc-pVDZ/cc-pVDZ-RI containing 4375 atomic orbitals and 14 700 auxiliary basis functions in ∼0.85 h. In the EFMO framework, the GPU RI-MP2 component shows near linear scaling for a large number of V100s when computing the energy of an 1800-atom mesoporous silica nanoparticle in a bath of 4000 water molecules. The parallel efficiencies of the GPU RI-MP2 component with 2304 and 4608 V100s are 98.0% and 96.1%, respectively.
REFERENCES
1.
D. G.
Fedorov
and K.
Kitaura
, The Fragment Molecular Orbital Method: Practical Applications to Large Molecular Systems
, 1st ed.
, edited by D. G.
Fedorov
and K.
Kitaura
(CRC Press
, Boca Raton
, 2009
).2.
D. G.
Fedorov
and K.
Kitaura
, “Theoretical development of the fragment molecular orbital (FMO) method
,” in Modern Methods for Theoretical Physical Chemistry of Biopolymers
(Elsevier Science
, 2006
), pp. 3
–38
.3.
M. S.
Gordon
, D. G.
Fedorov
, S. R.
Pruitt
, and L. V.
Slipchenko
, “Fragmentation methods: A route to accurate calculations on large systems
,” Chem. Rev.
112
, 632
–672
(2012
).4.
K.
Kitaura
, T.
Sawai
, T.
Asada
, T.
Nakano
, and M.
Uebayasi
, “Pair interaction molecular orbital method: An approximate computational method for molecular interactions
,” Chem. Phys. Lett.
312
, 319
–324
(1999
).5.
K.
Kitaura
, E.
Ikeo
, T.
Asada
, T.
Nakano
, and M.
Uebayasi
, “Fragment molecular orbital method: An approximate computational method for large molecules
,” Chem. Phys. Lett.
313
, 701
–706
(1999
).6.
T.
Nakano
, T.
Kaminuma
, T.
Sato
, Y.
Akiyama
, M.
Uebayasi
, and K.
Kitaura
, “Fragment molecular orbital method: Application to polypeptides
,” Chem. Phys. Lett.
318
, 614
–618
(2000
).7.
T.
Nakano
, T.
Kaminuma
, T.
Sato
, K.
Fukuzawa
, Y.
Akiyama
, M.
Uebayasi
, and K.
Kitaura
, “Fragment molecular orbital method: Use of approximate electrostatic potential
,” Chem. Phys. Lett.
351
, 475
–480
(2002
).8.
T.
Nagata
, D. G.
Fedorov
, and K.
Kitaura
, “Mathematical formulation of the fragment molecular orbital method BT
,” in Linear-Scaling Techniques in Computational Chemistry and Physics: Methods and Applications
, edited by R.
Zalesny
, M. G.
Papadopoulos
, P. G.
Mezey
, and J.
Leszczynski
(Springer Netherlands
, Dordrecht
, 2011
), pp. 17
–64
.9.
Y.
Komeiji
, Y.
Mochizuki
, and T.
Nakano
, “Three-body expansion and generalized dynamic fragmentation improve the fragment molecular orbital-based molecular dynamics (FMO-MD)
,” Chem. Phys. Lett.
484
, 380
–386
(2010
).10.
C.
Steinmann
, D. G.
Fedorov
, and J. H.
Jensen
, “Mapping enzymatic catalysis using the effective fragment molecular orbital method: Towards all ab initio biochemistry
,” PLoS One
8
, e60602
(2013
).11.
A. S.
Christensen
, C.
Steinmann
, D. G.
Fedorov
, and J. H.
Jensen
, “Hybrid RHF/MP2 geometry optimizations with the effective fragment molecular orbital method
,” PLoS One
9
, e88800
(2014
).12.
S. R.
Pruitt
, C.
Steinmann
, J. H.
Jensen
, and M. S.
Gordon
, “Fully integrated effective fragment molecular orbital method
,” J. Chem. Theory Comput.
9
, 2235
–2249
(2013
).13.
C.
Steinmann
, D. G.
Fedorov
, and J. H.
Jensen
, “The effective fragment molecular orbital method for fragments connected by covalent bonds
,” PLoS One
7
, e41117
(2012
).14.
C.
Steinmann
, D. G.
Fedorov
, and J. H.
Jensen
, “Effective fragment molecular orbital method: A merger of the effective fragment potential and fragment molecular orbital methods
,” J. Phys. Chem. A
114
, 8705
–8712
(2010
).15.
C.
Bertoni
and M. S.
Gordon
, “Analytic gradients for the effective fragment molecular orbital method
,” J. Chem. Theory Comput.
12
, 4743
–4767
(2016
).16.
C.
Møller
and M. S.
Plesset
, “Note on an approximation treatment for many-electron systems
,” Phys. Rev.
46
, 618
–622
(1934
).17.
J.
Čížek
, “On the correlation problem in atomic and molecular systems. Calculation of wavefunction components in Ursell‐type expansion using quantum‐field theoretical methods
,” J. Chem. Phys.
45
, 4256
–4266
(1966
).18.
I.
Røeggen
and E.
Wisløff-Nilssen
, “On the Beebe-Linderberg two-electron integral approximation
,” Chem. Phys. Lett.
132
, 154
–160
(1986
).19.
H.
Koch
, A.
Sánchez de Merás
, and T. B.
Pedersen
, “Reduced scaling in electronic structure calculations using Cholesky decompositions
,” J. Chem. Phys.
118
, 9481
–9484
(2003
).20.
N. H. F.
Beebe
and J.
Linderberg
, “Simplifications in the generation and transformation of two‐electron integrals in molecular calculations
,” Int. J. Quantum Chem.
12
, 683
–705
(1977
).21.
B.
Peng
and K.
Kowalski
, “Highly efficient and scalable compound decomposition of two-electron integral tensor and its application in coupled cluster calculations
,” J. Chem. Theory Comput.
13
, 4179
–4192
(2017
).22.
J. L.
Whitten
, “Coulombic potential energy integrals and approximations
,” J. Chem. Phys.
58
, 4496
–4501
(1973
).23.
M.
Feyereisen
, G.
Fitzgerald
, and A.
Komornicki
, “Use of approximate integrals in ab initio theory. An application in MP2 energy calculations
,” Chem. Phys. Lett.
208
, 359
–363
(1993
).24.
O.
Vahtras
, J.
Almlöf
, and M. W.
Feyereisen
, “Integral approximations for LCAO-SCF calculations
,” Chem. Phys. Lett.
213
, 514
–518
(1993
).25.
F.
Weigend
and M.
Häser
, “RI-MP2: First derivatives and global consistency
,” Theor. Chem. Acc.
97
, 331
–340
(1997
).26.
F.
Weigend
, M.
Häser
, H.
Patzelt
, and R.
Ahlrichs
, “RI-MP2: Optimized auxiliary basis sets and demonstration of efficiency
,” Chem. Phys. Lett.
294
, 143
–152
(1998
).27.
B. Q.
Pham
and M. S.
Gordon
, “Hybrid distributed/shared memory model for the RI-MP2 method in the fragment molecular orbital framework
,” J. Chem. Theory Comput.
15
, 5252
–5258
(2019
).28.
M.
Katouda
and T.
Nakajima
, “MPI/OpenMP hybrid parallel algorithm for resolution of identity second‐order Møller–Plesset perturbation calculation of analytical energy gradient for massively parallel multicore supercomputers
,” J. Comput. Chem.
38
, 489
–507
(2017
).29.
H.-J.
Werner
, F. R.
Manby
, and P. J.
Knowles
, “Fast linear scaling second-order Møller-Plesset perturbation theory (MP2) using local and density fitting approximations
,” J. Chem. Phys.
118
, 8149
–8160
(2003
).30.
B. Q.
Pham
and M. S.
Gordon
, “Compressing the four-index two-electron repulsion integral matrix using the resolution-of-the-identity approximation combined with the rank factorization approximation
,” J. Chem. Theory Comput.
15
, 2254
–2264
(2019
).31.
E. G.
Hohenstein
, R. M.
Parrish
, and T. J.
Martínez
, “Tensor hypercontraction density fitting. I. Quartic scaling second- and third-order Møller-Plesset perturbation theory
,” J. Chem. Phys.
137
, 044103
(2012
).32.
E. G.
Hohenstein
, S. I. L.
Kokkila
, R. M.
Parrish
, and T. J.
Martínez
, “Quartic scaling second-order approximate coupled cluster singles and doubles via tensor hypercontraction: THC-CC2
,” J. Chem. Phys.
138
, 124111
(2013
).33.
E. G.
Hohenstein
, S. I. L.
Kokkila
, R. M.
Parrish
, and T. J.
Martínez
, “Tensor hypercontraction equation-of-motion second-order approximate coupled cluster: Electronic excitation energies in O(N4) time
,” J. Phys. Chem. B
117
, 12972
–12978
(2013
).34.
E. G.
Hohenstein
, R. M.
Parrish
, C. D.
Sherrill
, and T. J.
Martínez
, “Communication: Tensor hypercontraction. III. Least-squares tensor hypercontraction for the determination of correlated wavefunctions
,” J. Chem. Phys.
137
, 221101
(2012
).35.
R. M.
Parrish
, E. G.
Hohenstein
, T. J.
Martínez
, and C. D.
Sherrill
, “Tensor hypercontraction. II. Least-squares renormalization
,” J. Chem. Phys.
137
, 224106
(2012
).36.
B. Q.
Pham
, D.
Datta
, and M. S.
Gordon
, “PDG: A composite method based on the resolution of the identity
,” J. Phys. Chem. A
125
, 9421
–9429
(2021
).37.
D.
Datta
and M. S.
Gordon
, “A massively parallel implementation of the CCSD(T) method using the resolution-of-the-identity approximation and a hybrid distributed/shared memory parallelization model
,” J. Chem. Theory Comput.
17
, 4799
–4822
(2021
).38.
B. Q.
Pham
and M. S.
Gordon
, “Development of the FMO/RI-MP2 fully analytic gradient using a hybrid-distributed/shared memory programming model
,” J. Chem. Theory Comput.
16
, 1039
–1054
(2020
).39.
M. S.
Gordon
, J. S.
Binkley
, J. A.
Pople
, W. J.
Pietro
, and W. J.
Hehre
, “Self-consistent molecular-orbital methods. 22. Small split-valence basis sets for second-row elements
,” J. Am. Chem. Soc.
104
, 2797
–2803
(1982
).40.
W. J.
Hehre
, R.
Ditchfield
, and J. A.
Pople
, “Self-consistent molecular orbital methods. XII. Further extensions of Gaussian-type basis sets for use in molecular orbital studies of organic molecules
,” J. Chem. Phys.
56
, 2257
–2261
(1972
).41.
L.
Roskop
, D. G.
Fedorov
, and M. S.
Gordon
, “Diffusion energy profiles in silica mesoporous molecular sieves modelled with the fragment molecular orbital method
,” Mol. Phys.
111
, 1622
–1629
(2013
).42.
B.
Pham
, M.
Alkan
, and M.
Gordon
, “Porting fragmentation methods to graphical processing units using an OpenMP application programming interface: Offloading the Fock build for low angular momentum functions
,” J. Chem. Theory Comput.
(published online 2023).43.
M.
Alkan
, B. Q.
Pham
, J. R.
Hammond
, and M. S.
Gordon
, “Enabling Fortran standard parallelism in GAMESS for accelerated quantum chemistry calculations
,” J. Chem. Theory Comput.
(submitted).44.
L.
Vogt
, R.
Olivares-Amaya
, S.
Kermes
, Y.
Shao
, C.
Amador-Bedolla
, and A.
Aspuru-Guzik
, “Accelerating resolution-of-the-identity second-order Møller–Plesset quantum chemistry calculations with graphical processing units
,” J. Phys. Chem. A
112
, 2049
–2057
(2008
).45.
G. M. J.
Barca
, C.
Snowdon
, J. L. G.
Vallejo
, F.
Kazemian
, A. P.
Rendell
, and M. S.
Gordon
, “Scaling correlated fragment molecular orbital calculations on Summit
,” in SC22: International Conference for High Performance Computing, Networking, Storage and Analysis
(IEEE Press
, 2022
), pp. 1
–14
.46.
J. J.
Eriksen
, “Chapter 12—Incrementally accelerating the RI-MP2 correlated method of electronic structure theory using OpenACC compiler directives
,” in Parallel Programming with OpenACC
, edited by R.
Farber
(Morgan Kaufmann
, Boston
, 2017
), pp. 241
–265
.47.
P. N.
Day
, J. H.
Jensen
, M. S.
Gordon
, S. P.
Webb
, W. J.
Stevens
, M.
Krauss
, D.
Garmer
, H.
Basch
, and D.
Cohen
, “An effective fragment method for modeling solvent effects in quantum mechanical calculations
,” J. Chem. Phys.
105
, 1968
–1986
(1996
).48.
M.
Katouda
and T.
Nakajima
, “MPI/OpenMP hybrid parallel algorithm of resolution of identity second-order Møller–Plesset perturbation calculation for massively parallel multicore supercomputers
,” J. Chem. Theory Comput.
9
, 5373
–5380
(2013
).49.
M.
Katouda
, A.
Naruse
, Y.
Hirano
, and T.
Nakajima
, “Massively parallel algorithm and implementation of RI‐MP2 energy calculation for peta‐scale many‐core supercomputers
,” J. Comput. Chem.
37
, 2623
–2633
(2016
).50.
M.
Katouda
, “Application of resolution of identity approximation of second-order Møller–Plesset perturbation theory to three-body fragment molecular orbital method
,” Theor. Chem. Acc.
130
, 449
–453
(2011
).51.
M.
Katouda
and S.
Nagase
, “Application of second-order Møller–Plesset perturbation theory with resolution-of-identity approximation to periodic systems
,” J. Chem. Phys.
133
, 184103
(2010
).52.
See https://docs.nvidia.com/cuda/cublas/index.html for CUDA Toolkit Documentation.
53.
R.
van der Pas
, E.
Stotzer
, and C.
Terboven
, Using OpenMP—The Next Step: Affinity, Accelerators, Tasking, and SIMD
(MIT Press
, 2017
).54.
See https://fortranwiki.org/fortran/show/Interoperability for Fortran Wiki: Interoperability.
55.
J.
Kwack
, C.
Bertoni
, B.
Pham
, and J.
Larkin
, “Performance of the RI-MP2 Fortran kernel of GAMESS on GPUs via directive-based offloading with math libraries
,” in Accelerator Programming Using Directives
, edited by S.
Wienke
and S.
Bhalachandra
(Springer International Publishing
, Cham
, 2020
), pp. 91
–113
.56.
N.
Frederick
, Cn Fullerene, https://nanotube.msu.edu/fullerene/fullerene-isomers.html, 2013.57.
See https://docs.olcf.ornl.gov/systems/summit_user_guide.html#nvidia-tesla-v100 for Summit user guide.
© 2023 Author(s). Published under an exclusive license by AIP Publishing.
2023
Author(s)
You do not currently have access to this content.
