We report a Kohn–Sham density functional theory calculation of a system with more than 200 000 atoms and 800 000 electrons using a real-space high-order finite-difference method to investigate the electronic structure of large spherical silicon nanoclusters. Our system of choice was a 20 nm large spherical nanocluster with 202 617 silicon atoms and 13 836 hydrogen atoms used to passivate the dangling surface bonds. To speed up the convergence of the eigenspace, we utilized Chebyshev-filtered subspace iteration, and for sparse matrix–vector multiplications, we used blockwise Hilbert space-filling curves, implemented in the PARSEC code. For this calculation, we also replaced our orthonormalization + Rayleigh–Ritz step with a generalized eigenvalue problem step. We utilized all of the 8192 nodes (458 752 processors) on the Frontera machine at the Texas Advanced Computing Center. We achieved two Chebyshev-filtered subspace iterations, yielding a good approximation of the electronic density of states. Our work pushes the limits on the capabilities of the current electronic structure solvers to nearly 106 electrons and demonstrates the potential of the real-space approach to efficiently parallelize large calculations on modern high-performance computing platforms.

1.
P.
Hohenberg
and
W.
Kohn
,
Phys. Rev.
136
,
B864
(
1964
).
2.
W.
Kohn
and
L. J.
Sham
,
Phys. Rev.
140
,
A1133
(
1965
).
3.
J. R.
Chelikowsky
and
M. L.
Cohen
, “
Ab initio pseudopotentials and the structural properties of semiconductors
,” in
Basic Properties of Semiconductors
,
Handbook on Semiconductors
(
Elsevier
,
1992
) pp.
59
112
; available at https://www.elsevier.com/books/basic-properties-of-semiconductors/landsberg/978-0-444-88855-6.
4.
J. R.
Chelikowsky
,
N.
Troullier
, and
Y.
Saad
,
Phys. Rev. Lett.
72
,
1240
(
1994
).
5.
A.
Stathopoulos
,
S.
Ogut
,
Y.
Saad
,
J.
Chelikowsky
, and
H.
Kim
,
Comput. Sci. Eng.
2
,
19
(
2000
).
6.
J. R.
Chelikowsky
,
L.
Kronik
,
I.
Vasiliev
,
M.
Jain
, and
Y.
Saad
, “
Using real space pseudopotentials for the electronic structure problem
,” in
Handbook of Numerical Analysis
,
Special Volume, Computational Chemistry Vol. 10
(
Elsevier
,
2003
), pp.
613
637
.
7.
M. M. G.
Alemany
,
M.
Jain
,
L.
Kronik
, and
J. R.
Chelikowsky
,
Phys. Rev. B
69
,
075101
(
2004
).
8.
L.
Kronik
,
A.
Makmal
,
M. L.
Tiago
,
M. M. G.
Alemany
,
M.
Jain
,
X.
Huang
,
Y.
Saad
, and
J. R.
Chelikowsky
,
Phys. Status Solidi B
243
,
1063
(
2006
).
9.
V.
Gavini
,
S.
Baroni
,
V.
Blum
,
D. R.
Bowler
,
A.
Buccheri
,
J. R.
Chelikowsky
,
S.
Das
,
W.
Dawson
,
P.
Delugas
,
M.
Dogan
,
C.
Draxl
,
G.
Galli
,
L.
Genovese
,
P.
Giannozzi
,
M.
Giantomassi
,
X.
Gonze
,
M.
Govoni
,
A.
Gulans
,
F.
Gygi
,
J. M.
Herbert
,
S.
Kokott
,
T. D.
Kühne
,
K.-H.
Liou
,
T.
Miyazaki
,
P.
Motamarri
,
A.
Nakata
,
J. E.
Pask
,
C.
Plessl
,
L. E.
Ratcliff
,
R. M.
Richard
,
M.
Rossi
,
R.
Schade
,
M.
Scheffler
,
O.
Schütt
,
P.
Suryanarayana
,
M.
Torrent
,
L.
Truflandier
,
T. L.
Windus
,
Q.
Xu
,
V. W.-Z.
Yu
, and
D.
Perez
, “
Roadmap on electronic structure codes in the exascale era
,” arXiv:2209.12747 [cond-mat, physics:physics] (
2022
).
10.
J.-I.
Iwata
,
D.
Takahashi
,
A.
Oshiyama
,
T.
Boku
,
K.
Shiraishi
,
S.
Okada
, and
K.
Yabana
,
J. Comput. Phys.
229
,
2339
(
2010
).
11.
X.
Andrade
,
D.
Strubbe
,
U.
De Giovannini
,
A. H.
Larsen
,
M. J. T.
Oliveira
,
J.
Alberdi-Rodriguez
,
A.
Varas
,
I.
Theophilou
,
N.
Helbig
,
M. J.
Verstraete
,
L.
Stella
,
F.
Nogueira
,
A.
Aspuru-Guzik
,
A.
Castro
,
M. A. L.
Marques
, and
A.
Rubio
,
Phys. Chem. Chem. Phys.
17
,
31371
(
2015
).
12.
W.
Mi
,
X.
Shao
,
C.
Su
,
Y.
Zhou
,
S.
Zhang
,
Q.
Li
,
H.
Wang
,
L.
Zhang
,
M.
Miao
,
Y.
Wang
, and
Y.
Ma
,
Comput. Phys. Commun.
200
,
87
(
2016
).
13.
V.
Michaud-Rioux
,
L.
Zhang
, and
H.
Guo
,
J. Comput. Phys.
307
,
593
(
2016
).
14.
S.
Ghosh
and
P.
Suryanarayana
,
Comput. Phys. Commun.
212
,
189
(
2017
).
15.
S.
Ghosh
and
P.
Suryanarayana
,
Comput. Phys. Commun.
216
,
109
(
2017
).
16.
J.
Bernholc
,
M.
Hodak
, and
W.
Lu
,
J. Phys.: Condens. Matter
20
,
294205
(
2008
).
17.
O.
Cohen
,
L.
Kronik
, and
A.
Brandt
,
J. Chem. Theory Comput.
9
,
4744
(
2013
).
18.
J.
Zhang
,
Y.
Cheng
,
W.
Lu
,
E.
Briggs
,
A. J.
Ramirez-Cuesta
, and
J.
Bernholc
,
J. Chem. Theory Comput.
15
,
6859
(
2019
).
19.
S. R.
Jensen
,
S.
Saha
,
J. A.
Flores-Livas
,
W.
Huhn
,
V.
Blum
,
S.
Goedecker
, and
L.
Frediani
,
J. Phys. Chem. Lett.
8
,
1449
(
2017
).
20.
J. E.
Pask
,
B. M.
Klein
,
P. A.
Sterne
, and
C. Y.
Fong
,
Comput. Phys. Commun.
135
,
1
(
2001
).
21.
B.
Kanungo
and
V.
Gavini
,
Phys. Rev. B
95
,
035112
(
2017
).
22.
P.
Motamarri
,
S.
Das
,
S.
Rudraraju
,
K.
Ghosh
,
D.
Davydov
, and
V.
Gavini
,
Comput. Phys. Commun.
246
,
106853
(
2020
).
23.
S.
Das
,
P.
Motamarri
,
V.
Subramanian
,
D. M.
Rogers
, and
V.
Gavini
,
Comput. Phys. Commun.
280
,
108473
(
2022
).
24.
J. R.
Chelikowsky
,
J. Phys. D: Appl. Phys.
33
,
R33
(
2000
).
25.
Y.
Saad
,
J. R.
Chelikowsky
, and
S. M.
Shontz
,
SIAM Rev.
52
,
3
(
2010
).
26.
G.
Schofield
,
J. R.
Chelikowsky
, and
Y.
Saad
,
Comput. Phys. Commun.
183
,
497
(
2012
).
27.
A. S.
Banerjee
,
L.
Lin
,
P.
Suryanarayana
,
C.
Yang
, and
J. E.
Pask
,
J. Chem. Theory Comput.
14
,
2930
(
2018
).
28.
K.-H.
Liou
,
C.
Yang
, and
J. R.
Chelikowsky
,
Comput. Phys. Commun.
254
,
107330
(
2020
).
29.
K.-H.
Liou
,
A.
Biller
,
L.
Kronik
, and
J. R.
Chelikowsky
,
J. Chem. Theory Comput.
17
,
4039
(
2021
).
30.
J. K.
Lawder
and
P. J. H.
King
, in
Advances in Databases
,
Lecture Notes in Computer Science
, edited by
B.
Lings
and
K.
Jeffery
(
Springer
,
Berlin, Heidelberg
,
2000
), pp.
20
35
.
31.
B.
Moon
,
H. V.
Jagadish
,
C.
Faloutsos
, and
J. H.
Saltz
,
IEEE Trans. Knowl. Data Eng.
13
,
124
(
2001
).
32.
A.-J. N.
Yzelman
and
D.
Roose
,
IEEE Trans. Parallel Distrib. Syst.
25
,
116
(
2014
).
33.
P.
Xu
and
S.
Tirthapura
,
ACM Trans. Database Syst.
39
(
10
),
1
(
2014
).
34.
D. M.
Ceperley
and
B. J.
Alder
,
Phys. Rev. Lett.
45
,
566
(
1980
).
35.
J. P.
Perdew
and
A.
Zunger
,
Phys. Rev. B
23
,
5048
(
1981
).
36.
N.
Troullier
and
J. L.
Martins
,
Phys. Rev. B
43
,
1993
(
1991
).
37.
L.
Kleinman
and
D. M.
Bylander
,
Phys. Rev. Lett.
48
,
1425
(
1982
).
38.
M.
Dogan
,
K.-H.
Liou
, and
J. R.
Chelikowsky
, “
Solving the electronic structure problem for over 100,000 atoms in real-space
,” arXiv:2303.00790 [cond-mat, physics:physics] (
2023
).
39.
P.
Kus
,
A.
Marek
,
S. S.
Kocher
,
H. H.
Kowalski
,
C.
Carbogno
,
C.
Scheurer
,
K.
Reuter
,
M.
Scheffler
, and
H.
Lederer
,
Parallel Comput.
85
,
167
(
2019
).
40.
A.
Natan
,
A.
Benjamini
,
D.
Naveh
,
L.
Kronik
,
M. L.
Tiago
,
S. P.
Beckman
, and
J. R.
Chelikowsky
,
Phys. Rev. B
78
,
075109
(
2008
).
41.
H. J.
Monkhorst
and
J. D.
Pack
,
Phys. Rev. B
13
,
5188
(
1976
).
42.
M. L.
Cohen
and
J. R.
Chelikowsky
,
Electronic Structure and Optical Properties of Semiconductors
,
Springer Series in Solid-State Sciences Vol. 75
(
Springer Berlin Heidelberg
,
1988
).
You do not currently have access to this content.