We introduce an extension of the divide-and-conquer (DC) algorithmic paradigm called divide-conquer-recombine (DCR) to perform large quantum molecular dynamics (QMD) simulations on massively parallel supercomputers, in which interatomic forces are computed quantum mechanically in the framework of density functional theory (DFT). In DCR, the DC phase constructs globally informed, overlapping local-domain solutions, which in the recombine phase are synthesized into a global solution encompassing large spatiotemporal scales. For the DC phase, we design a lean divide-and-conquer (LDC) DFT algorithm, which significantly reduces the prefactor of the O(N) computational cost for N electrons by applying a density-adaptive boundary condition at the peripheries of the DC domains. Our globally scalable and locally efficient solver is based on a hybrid real-reciprocal space approach that combines: (1) a highly scalable real-space multigrid to represent the global charge density; and (2) a numerically efficient plane-wave basis for local electronic wave functions and charge density within each domain. Hybrid space-band decomposition is used to implement the LDC-DFT algorithm on parallel computers. A benchmark test on an IBM Blue Gene/Q computer exhibits an isogranular parallel efficiency of 0.984 on 786 432 cores for a 50.3 × 106-atom SiC system. As a test of production runs, LDC-DFT-based QMD simulation involving 16 661 atoms is performed on the Blue Gene/Q to study on-demand production of hydrogen gas from water using LiAl alloy particles. As an example of the recombine phase, LDC-DFT electronic structures are used as a basis set to describe global photoexcitation dynamics with nonadiabatic QMD (NAQMD) and kinetic Monte Carlo (KMC) methods. The NAQMD simulations are based on the linear response time-dependent density functional theory to describe electronic excited states and a surface-hopping approach to describe transitions between the excited states. A series of techniques are employed for efficiently calculating the long-range exact exchange correction and excited-state forces. The NAQMD trajectories are analyzed to extract the rates of various excitonic processes, which are then used in KMC simulation to study the dynamics of the global exciton flow network. This has allowed the study of large-scale photoexcitation dynamics in 6400-atom amorphous molecular solid, reaching the experimental time scales.

1.
L.
Greengard
and
V.
Rokhlin
,
J. Comput. Phys.
73
(
2
),
325
348
(
1987
).
2.
C. A.
White
and
M.
Headgordon
,
J. Chem. Phys.
101
(
8
),
6593
6605
(
1994
).
3.
A.
Nakano
,
R. K.
Kalia
, and
P.
Vashishta
,
Comput. Phys. Commun.
83
(
2–3
),
197
214
(
1994
).
4.
S.
Ogata
,
T. J.
Campbell
,
R. K.
Kalia
,
A.
Nakano
,
P.
Vashishta
, and
S.
Vemparala
,
Comput. Phys. Commun.
153
(
3
),
445
461
(
2003
).
5.
J. J. M.
Cuppen
,
Numer. Math.
36
(
2
),
177
195
(
1981
).
6.
W. N.
Gansterer
,
R. C.
Ward
, and
R. P.
Muller
,
ACM Trans. Math. Software
28
(
1
),
45
58
(
2002
).
7.
A.
Nakano
,
Comput. Phys. Commun.
104
(
1–3
),
59
69
(
1997
).
8.
W. T.
Yang
,
Phys. Rev. Lett.
66
(
11
),
1438
1441
(
1991
).
9.
S. L.
Dixon
and
K. M.
Merz
,
J. Chem. Phys.
107
,
879
(
1997
).
10.
F.
Shimojo
,
R. K.
Kalia
,
A.
Nakano
, and
P.
Vashishta
,
Comput. Phys. Commun.
167
(
3
),
151
164
(
2005
).
11.
T.
Ozaki
,
Phys. Rev. B
74
(
24
),
245101
(
2006
).
12.
M.
Kobayashi
and
H.
Nakai
,
J. Chem. Phys.
129
(
4
),
044103
(
2008
).
13.
F.
Shimojo
,
R. K.
Kalia
,
A.
Nakano
, and
P.
Vashishta
,
Phys. Rev. B
77
(
8
),
085103
(
2008
).
14.
N.
Ohba
,
S.
Ogata
,
T.
Kouno
,
T.
Tanmura
, and
R.
Kobayashi
,
Comput. Phys. Commun.
183
(
8
),
1664
1673
(
2012
).
15.
An exaflop/s computer can operate 1018 arithmetic operations per second.
16.
K.
Nomura
,
H.
Dursun
,
R.
Seymour
,
W.
Wang
,
R. K.
Kalia
,
A.
Nakano
,
P.
Vashishta
,
F.
Shimojo
, and
L. H.
Yang
,
Proceedings of the International Parallel and Distributed Processing Symposium
, IPDPS 2009 (
IEEE
,
2009
).
17.
P.
Hohenberg
and
W.
Kohn
,
Phys. Rev.
136
(
3
),
B864
B871
(
1964
).
18.
F.
Gygi
,
E.
Draeger
,
B. R.
de Supinski
,
R. K.
Yates
,
F.
Franchetti
,
S.
Kral
,
J.
Lorenz
,
C. W.
Ueberhuber
,
J. A.
Gunneis
, and
J. C.
Sexton
,
Proceedings of Supercomputing
, SC05 (
ACM/IEEE
,
2005
).
19.
R.
Car
and
M.
Parrinello
,
Phys. Rev. Lett.
55
,
2471
2474
(
1985
).
20.
M. C.
Payne
,
M. P.
Teter
,
D. C.
Allan
,
T. A.
Arias
, and
J. D.
Joannopoulos
,
Rev. Mod. Phys.
64
(
4
),
1045
1097
(
1992
).
21.
D. R.
Bowler
and
T.
Miyazaki
,
Rep. Prog. Phys.
75
(
3
),
036503
(
2012
).
22.
W.
Kohn
,
Phys. Rev. Lett.
76
(
17
),
3168
3171
(
1996
).
23.
E.
Prodan
and
W.
Kohn
,
Proc. Natl. Acad. Sci. U.S.A.
102
(
33
),
11635
11638
(
2005
).
24.
M.
Benzi
,
P.
Boito
, and
N.
Razouk
,
SIAM Rev.
55
(
1
),
3
64
(
2013
).
25.
A.
Nakano
,
R. K.
Kalia
,
K.
Nomura
,
A.
Sharma
,
P.
Vashishta
,
F.
Shimojo
,
A. C. T.
van Duin
,
W. A.
Goddard
,
R.
Biswas
,
D.
Srivastava
, and
L. H.
Yang
,
Intl. J. High Performance Comput. Appl.
22
(
1
),
113
128
(
2008
).
26.
A related O(N) algorithm called density fragment interaction has been developed for large molecular systems by
Weitao
Yan
and collaborators:
X. Q.
Hu
,
Y. D.
Jin
,
X. C.
Zeng
,
H.
Hu
, and
W. T.
Yang
,
Phys. Chem. Chem. Phys.
14
(
21
),
7700
7709
(
2012
).
27.
T. V. T.
Duy
and
T.
Ozaki
,
Comput. Phys. Commun.
185
(
3
),
777
789
(
2014
).
28.
M. J.
Cawkwell
and
A. M. N.
Niklasson
,
J. Chem. Phys.
137
(
13
),
134105
(
2012
).
29.
F.
Shimojo
,
A.
Nakano
,
R. K.
Kalia
, and
P.
Vashishta
,
Appl. Phys. Lett.
95
(
4
),
043114
(
2009
).
30.
Y.
Okiyama
,
T.
Tsukamoto
,
C.
Watanabe
,
K.
Fukuzawa
,
S.
Tanaka
, and
Y.
Mochizuki
,
Chem. Phys. Lett.
566
,
25
31
(
2013
).
31.
S.
Tanaka
,
C.
Watanabe
, and
Y.
Okiyama
,
Chem. Phys. Lett.
556
,
272
277
(
2013
).
32.
S.
Tsuneyuki
,
T.
Kobori
,
K.
Akagi
,
K.
Sodeyama
,
K.
Terakura
, and
H.
Fukuyama
,
Chem. Phys. Lett.
476
(
1–3
),
104
108
(
2009
).
33.
T.
Kobori
,
K.
Sodeyama
,
T.
Otsuka
,
Y.
Tateyama
, and
S.
Tsuneyuki
,
J. Chem. Phys.
139
(
9
),
094113
(
2013
).
34.
C.
Gollub
,
S.
Avdoshenko
,
R.
Gutierrez
,
Y.
Berlin
, and
G.
Cuniberti
,
Isr. J. Chem.
52
(
5
),
452
460
(
2012
).
35.
H.
Kitoh-Nishioka
and
K.
Ando
,
J. Phys. Chem. B
116
(
43
),
12933
12945
(
2012
).
36.
W.
Mou
,
S.
Hattori
,
P.
Rajak
,
F.
Shimojo
, and
A.
Nakano
,
Appl. Phys. Lett.
102
(
17
),
173301
(
2013
).
37.
S. T.
Roberts
,
R. E.
McAnally
,
J. N.
Mastron
,
D. H.
Webber
,
M. T.
Whited
,
R. L.
Brutchey
,
M. E.
Thompson
, and
S. E.
Bradforth
,
J. Am. Chem. Soc.
134
(
14
),
6388
6400
(
2012
).
38.
M. B.
Smith
and
J.
Michl
,
Ann. Rev. Phys. Chem.
64
,
361
386
(
2013
).
39.
C. F.
Craig
,
W. R.
Duncan
, and
O. V.
Prezhdo
,
Phys. Rev. Lett.
95
(
16
),
163001
(
2005
).
40.
C. P.
Hu
,
H.
Hirai
, and
O.
Sugino
,
J. Chem. Phys.
127
(
6
),
064103
(
2007
).
41.
E.
Tapavicza
,
I.
Tavernelli
, and
U.
Rothlisberger
,
Phys. Rev. Lett.
98
(
2
),
023001
(
2007
).
42.
X.
Zhang
,
Z.
Li
, and
G.
Lu
,
Phys. Rev. B
84
(
23
),
235208
(
2011
).
43.
W.
Mou
,
S.
Ohmura
,
F.
Shimojo
, and
A.
Nakano
,
Appl. Phys. Lett.
100
(
20
),
203306
(
2012
).
44.
M. E.
Casida
, in
Recent Advances in Density Functional Methods (Part I)
, edited by
D. P.
Chong
(
World Scientific
,
Singapore
,
1995
), pp.
155
192
.
45.
J. C.
Tully
,
J. Chem. Phys.
93
(
2
),
1061
1071
(
1990
).
46.
J. R.
Schmidt
,
P. V.
Parandekar
, and
J. C.
Tully
,
J. Chem. Phys.
129
(
4
),
044104
(
2008
).
47.
O. V.
Prezhdo
,
J. Chem. Phys.
111
(
18
),
8366
8377
(
1999
).
48.
A. W.
Jasper
,
S. N.
Stechmann
, and
D. G.
Truhlar
,
J. Chem. Phys.
116
(
13
),
5424
5431
(
2002
).
49.
A. B.
Bortz
,
M. H.
Kalos
, and
J. L.
Lebowitz
,
J. Comput. Phys.
17
(
1
),
10
18
(
1975
).
50.
D. T.
Gillespie
,
J. Comput. Phys.
22
(
4
),
403
434
(
1976
).
51.
K. A.
Fichthorn
and
W. H.
Weinberg
,
J. Chem. Phys.
95
(
2
),
1090
1096
(
1991
).
52.
A. F.
Voter
, in
Radiation Effects in Solids
, edited by
K. E.
Sickafus
,
E. A.
Kotomin
, and
B. P.
Uberuaga
(
Springer
,
Dordrecht, The Netherlands
,
2006
).
53.
A.
Brandt
,
Math. Comput.
31
(
138
),
333
390
(
1977
).
54.
A.
Nakano
,
P.
Vashishta
, and
R. K.
Kalia
,
Comput. Phys. Commun.
83
(
2–3
),
181
196
(
1994
).
55.
M.
Kunaseth
,
R. K.
Kalia
,
A.
Nakano
,
K.
Nomura
, and
P.
Vashishta
,
Proceedings of Supercomputing
, SC13 (ACM/IEEE,
2013
).
56.
A.
Dreuw
,
J. L.
Weisman
, and
M.
Head-Gordon
,
J. Chem. Phys.
119
(
6
),
2943
2946
(
2003
).
57.
M.
Walter
,
H.
Hakkinen
,
L.
Lehtovaara
,
M.
Puska
,
J.
Enkovaara
,
C.
Rostgaard
, and
J. J.
Mortensen
,
J. Chem. Phys.
128
(
24
),
244101
(
2008
).
58.
M. E.
Casida
and
M.
Huix-Rotlant
,
Annu. Rev. Phys. Chem.
63
,
287
323
(
2012
).
59.
F.
Shimojo
,
S.
Ohmura
,
W.
Mou
,
R. K.
Kalia
,
A.
Nakano
, and
P.
Vashishta
,
Comput. Phys. Commun.
184
(
1
),
1
8
(
2013
).
60.
H. M.
Jaeger
,
S.
Fischer
, and
O. V.
Prezhdo
,
J. Chem. Phys.
137
(
22
),
22A545
(
2012
).
61.
J. P.
Perdew
,
K.
Burke
, and
M.
Ernzerhof
,
Phys. Rev. Lett.
77
(
18
),
3865
3868
(
1996
).
62.
Y.
Tawada
,
T.
Tsuneda
,
S.
Yanagisawa
,
T.
Yanai
, and
K.
Hirao
,
J. Chem. Phys.
120
(
18
),
8425
8433
(
2004
).
63.
X.
Zhang
,
Z.
Li
, and
G.
Lu
,
J. Phys.: Condens. Matter
24
(
20
),
205801
(
2012
).
64.
G. J.
Martyna
and
M. E.
Tuckerman
,
J. Chem. Phys.
110
(
6
),
2810
2821
(
1999
).
65.
J.
Heyd
,
G. E.
Scuseria
, and
M.
Ernzerhof
,
J. Chem. Phys.
118
(
18
),
8207
8215
(
2003
).
66.
I.
Tavernelli
,
B. F. E.
Curchod
,
A.
Laktionov
, and
U.
Rothlisberger
,
J. Chem. Phys.
133
(
19
),
194101
(
2010
).
67.
F.
Shimojo
,
S.
Ohmura
,
R. K.
Kalia
,
A.
Nakano
, and
P.
Vashishta
,
Phys. Rev. Lett.
104
(
12
),
126102
(
2010
).
68.
K.
Shimamura
,
F.
Shimojo
,
R. K.
Kalia
,
A.
Nakano
, and
P.
Vashishta
,
Phys. Rev. Lett.
111
(
6
),
066103
(
2013
).
69.
A. S.
Torralba
,
D. R.
Bowler
,
T.
Miyazaki
, and
M. J.
Gillan
,
J. Chem. Theory. Comput.
5
(
6
),
1499
1505
(
2009
).
70.
N.
Vukmirovic
and
L. W.
Wang
,
J. Chem. Phys.
134
(
9
),
094119
(
2011
).
71.
G. J. O.
Beran
and
S.
Hirata
,
Phys. Chem. Chem. Phys.
14
(
21
),
7559
7561
(
2012
).
72.
T. A.
Wesolowski
and
A.
Warshel
,
J. Phys. Chem.
97
(
30
),
8050
8053
(
1993
).
73.
N.
Govind
,
Y. A.
Wang
, and
E. A.
Carter
,
J. Chem. Phys.
110
(
16
),
7677
7688
(
1999
).
74.
P.
Elliott
,
K.
Burke
,
M. H.
Cohen
, and
A.
Wasserman
,
Phys. Rev. A
82
(
2
),
024501
(
2010
).
75.
A.
Nakano
and
S.
Ichimaru
,
Phys. Rev. B
39
(
8
),
4930
4937
(
1989
).
76.
M.
Frigo
and
S. G.
Johnson
,
Proc. IEEE
93
(
2
),
216
231
(
2005
).
77.
Y.
Shiihara
,
O.
Kuwazuru
, and
N.
Yoshikawa
,
Model Simul. Mater. Sci. Eng.
16
(
3
),
035004
(
2008
).
78.
E.
Martinez
,
J.
Marian
,
M. H.
Kalos
, and
J. M.
Perlado
,
J. Comput. Phys.
227
(
8
),
3804
3823
(
2008
).
79.
M.
Kunaseth
,
R. K.
Kalia
,
A.
Nakano
,
P.
Vashishta
,
D. F.
Richards
, and
J. N.
Glosli
,
Proceedings of the International Workshop on Parallel and Distributed Scientific and Engineering Computing
, PDSEC-13 (
IEEE
,
2013
).
80.
P. E.
Blochl
,
Phys. Rev. B
50
(
24
),
17953
17979
(
1994
).
81.
G.
Kresse
and
D.
Joubert
,
Phys. Rev. B
59
(
3
),
1758
1775
(
1999
).
82.
S. G.
Louie
,
S.
Froyen
, and
M. L.
Cohen
,
Phys. Rev. B
26
(
4
),
1738
1742
(
1982
).
83.
Y.
Fu
,
Y. H.
Zhou
,
H. B.
Su
,
F. Y. C.
Boey
, and
H.
Agren
,
J. Phys. Chem. C
114
(
9
),
3743
3747
(
2010
).
84.
I.
Szlufarska
,
A.
Nakano
, and
P.
Vashishta
,
Science
309
(
5736
),
911
914
(
2005
).
85.
H.
Chen
,
R. K.
Kalia
,
E.
Kaxiras
,
G.
Lu
,
A.
Nakano
,
K.
Nomura
,
A. C. T.
van Duin
,
P.
Vashishta
, and
Z.
Yuan
,
Phys. Rev. Lett.
104
(
15
),
155502
(
2010
).
86.
Z.
Yuan
and
A.
Nakano
,
Nano Lett.
13
(
10
),
4925
4930
(
2013
).
87.
A.
Warshel
and
M.
Karplus
,
J. Am. Chem. Soc.
94
(
16
),
5612
5625
(
1972
).
88.
A.
Warshel
and
M.
Levitt
,
J. Mol. Biol.
103
(
2
),
227
249
(
1976
).
89.
S.
Ogata
,
E.
Lidorikis
,
F.
Shimojo
,
A.
Nakano
,
P.
Vashishta
, and
R. K.
Kalia
,
Comput. Phys. Commun.
138
(
2
),
143
154
(
2001
).
90.
H.
Takemiya
,
Y.
Tanaka
,
S.
Sekiguchi
,
S.
Ogata
,
R. K.
Kalia
,
A.
Nakano
, and
P.
Vashishta
,
Proceedings of Supercomputing
, SC06 (
IEEE/ACM
,
2006
).
91.
S.
Dapprich
,
I.
Komáromi
,
K. S.
Byun
,
K.
Morokuma
, and
M. J.
Frisch
,
J. Mol. Struct.
461–462
,
1
21
(
1999
).
92.
A. C. T.
van Duin
,
S.
Dasgupta
,
F.
Lorant
, and
W. A.
Goddard
,
J. Phys. Chem. A
105
(
41
),
9396
9409
(
2001
).
93.
K.
Nomura
,
R. K.
Kalia
,
A.
Nakano
,
P.
Vashishta
,
A. C. T.
van Duin
, and
W. A.
Goddard
,
Phys. Rev. Lett.
99
(
14
),
148303
(
2007
).
94.
M.
Vedadi
,
A.
Choubey
,
K.
Nomura
,
R. K.
Kalia
,
A.
Nakano
,
P.
Vashishta
, and
A. C. T.
van Duin
,
Phys. Rev. Lett.
105
(
1
),
014503
(
2010
).
95.
S. B.
Sinnott
and
D. W.
Brenner
,
MRS Bull.
37
(
5
),
469
473
(
2012
).
96.
A.
Shekhar
,
K.
Nomura
,
R. K.
Kalia
,
A.
Nakano
, and
P.
Vashishta
,
Phys. Rev. Lett.
111
(
18
),
184503
(
2013
).
97.
C. E.
Wilmer
,
M.
Leaf
,
C. Y.
Lee
,
O. K.
Farha
,
B. G.
Hauser
,
J. T.
Hupp
, and
R. Q.
Snurr
,
Nat. Chem.
4
(
2
),
83
89
(
2012
).
98.
A. M.
Virshup
,
J.
Contreras-Garcia
,
P.
Wipf
,
W. T.
Yang
, and
D. N.
Beratan
,
J. Am. Chem. Soc.
135
(
19
),
7296
7303
(
2013
).
You do not currently have access to this content.