We describe a massively parallel implementation of the recently developed discontinuous Galerkin density functional theory (DGDFT) method, for efficient large-scale Kohn-Sham DFT based electronic structure calculations. The DGDFT method uses adaptive local basis (ALB) functions generated on-the-fly during the self-consistent field iteration to represent the solution to the Kohn-Sham equations. The use of the ALB set provides a systematic way to improve the accuracy of the approximation. By using the pole expansion and selected inversion technique to compute electron density, energy, and atomic forces, we can make the computational complexity of DGDFT scale at most quadratically with respect to the number of electrons for both insulating and metallic systems. We show that for the two-dimensional (2D) phosphorene systems studied here, using 37 basis functions per atom allows us to reach an accuracy level of 1.3 × 10−4 Hartree/atom in terms of the error of energy and 6.2 × 10−4 Hartree/bohr in terms of the error of atomic force, respectively. DGDFT can achieve 80% parallel efficiency on 128,000 high performance computing cores when it is used to study the electronic structure of 2D phosphorene systems with 3500-14 000 atoms. This high parallel efficiency results from a two-level parallelization scheme that we will describe in detail.

1.
P.
Hohenberg
and
W.
Kohn
,
Phys. Rev.
136
,
B864
(
1964
).
2.
W.
Kohn
and
L. J.
Sham
,
Phys. Rev.
140
,
B1133
(
1965
).
3.
S.
Goedecker
,
Rev. Mod. Phys.
71
,
1085
(
1999
).
4.
H.
Shang
,
H.
Xiang
,
Z.
Li
, and
J.
Yang
,
Int. Rev. Phys. Chem.
29
,
665
(
2010
).
5.
D. R.
Bowler
and
T.
Miyazaki
,
Rep. Prog. Phys.
75
,
036503
(
2012
).
6.
J. M.
Soler
,
E.
Artacho
,
J. D.
Gale
,
A.
García
,
J.
Junquera
,
P.
Ordejón
, and
D.
Sánchez-Portal
,
J. Phys.: Condens. Matter
14
,
2745
(
2002
).
7.
T.
Ozaki
and
H.
Kino
,
Phys. Rev. B
72
,
045121
(
2005
).
8.
X.
Qin
,
H.
Shang
,
H.
Xiang
,
Z.
Li
, and
J.
Yang
,
Int. J. Quantum Chem.
115
,
647
(
2014
).
9.
J.
VandeVondele
,
M.
Krack
,
F.
Mohamed
,
M.
Parrinello
,
T.
Chassaing
, and
J.
Hutter
,
Comput. Phys. Commun.
167
,
103
(
2005
).
10.
M. J.
Gillan
,
D. R.
Bowler
,
A. S.
Torralba
, and
T.
Miyazaki
,
Comput. Phys. Commun.
177
,
14
(
2007
).
11.
G.
Kresse
and
J.
Furthmüller
,
Phys. Rev. B
54
,
11169
(
1996
).
12.
G.
Kresse
and
J.
Hafner
,
Phys. Rev. B
47
,
558(R)
(
1993
).
13.
P.
Giannozzi
,
S.
Baroni
,
N.
Bonini
,
M.
Calandra
,
R.
Car
,
C.
Cavazzoni
,
D.
Ceresoli
,
G. L.
Chiarotti
,
M.
Cococcioni
,
I.
Dabo
 et al,
J. Phys.: Condens. Matter
21
,
395502
(
2009
).
14.
X.
Gonze
,
B.
Amadon
,
P.-M.
Anglade
,
J.-M.
Beuken
,
F.
Bottin
,
P.
Boulanger
,
F.
Bruneval
,
D.
Caliste
,
R.
Caracas
,
M.
Côté
 et al,
Comput. Phys. Commun.
180
,
2582
(
2009
).
15.
F.
Gygi
,
IBM J. Res. Dev.
52
,
1
(
2008
).
16.
J.
VandeVondele
,
U.
Bors̆tnik
, and
J.
Hutter
,
J. Chem. Theory Comput.
8
,
3565
(
2012
).
17.
D. R.
Bowler
and
T.
Miyazaki
,
J. Phys.: Condens. Matter
22
,
074207
(
2010
).
18.
C.
Saravanan
,
Y.
Shao
,
R.
Baer
,
P. N.
Ross
, and
M.
Head-Gordon
,
J. Comput. Chem.
24
,
618
(
2003
).
19.
E. H.
Rubensson
and
E.
Rudberg
,
J. Comput. Chem.
32
,
1411
(
2011
).
20.
U.
Bors̆tnik
,
J.
VandeVondele
,
V.
Weber
, and
J.
Hutter
,
Parallel Comput.
40
,
47
(
2014
).
21.
L.
Lin
,
J.
Lu
,
L.
Ying
, and
W.
E
,
J. Comput. Phys.
231
,
2140
(
2012
).
22.
D. N.
Arnold
,
SIAM J. Numer. Anal.
19
,
742
(
1982
).
23.
D. N.
Arnold
,
F.
Brezzi
,
B.
Cockburn
, and
L. D.
Marini
,
SIAM J. Numer. Anal.
39
,
1749
(
2002
).
24.
C. K.
Skylaris
,
P. D.
Haynes
,
A. A.
Mostofi
, and
M. C.
Payne
,
J. Chem. Phys.
122
,
084119
(
2005
).
25.
S.
Mohr
,
L. E.
Ratcliff
,
P.
Boulanger
,
L.
Genovese
,
D.
Caliste
,
T.
Deutsch
, and
S.
Goedecker
,
J. Chem. Phys.
140
,
204110
(
2014
).
26.
M. J.
Rayson
and
P. R.
Briddon
,
Phys. Rev. B
80
,
205104
(
2009
).
27.
J.
Kaye
,
L.
Lin
, and
C.
Yang
,
Commun. Math. Sci.
13
,
1741
(
2015
).
28.
L.
Lin
,
J.
Lu
,
L.
Ying
,
R.
Car
, and
W.
E
,
Commun. Math. Sci.
7
,
755
(
2009
).
29.
L.
Lin
,
M.
Chen
,
C.
Yang
, and
L.
He
,
J. Phys.: Condens. Matter
25
,
295501
(
2013
).
30.
L.
Lin
,
A.
García
,
G.
Huhs
, and
C.
Yang
,
J. Phys.: Condens. Matter
26
,
305503
(
2014
).
31.
G.
Zhang
,
L.
Lin
,
W.
Hu
,
C.
Yang
, and
J. E.
Pask
, “
Adaptive local basis set for Kohn-Sham density functional theory in a discontinuous Galerkin framework II: Force, vibration, and molecular dynamics calculations
” (published online 2015).
32.
L.
Kleinman
and
D. M.
Bylander
,
Phys. Rev. Lett.
48
,
1425
(
1982
).
33.
N. D.
Mermin
,
Phys. Rev.
137
,
A1441
(
1965
).
34.
L.
Lin
,
J.
Lu
,
L.
Ying
, and
W.
E
,
J. Chem. Phys.
231
,
4515
(
2012
).
35.
A. V.
Knyazev
,
SIAM J. Sci. Comput.
23
,
517
(
2001
).
36.
M. C.
Payne
,
M. P.
Teter
,
D. C.
Allen
,
T. A.
Arias
, and
J. D.
Joannopoulos
,
Rev. Mod. Phys.
64
,
1045
(
1992
).
37.
L. N.
Trefethen
,
Spectral Methods in MATLAB
(
SIAM
,
2000
), Vol.
10
.
38.
T.
Auckenthaler
,
V.
Blum
,
H. J.
Bungartz
,
T.
Huckle
,
R.
Johanni
,
L.
Krämer
,
B.
Lang
,
H.
Lederer
, and
P. R.
Willems
,
Parallel Comput.
37
,
783
(
2011
).
39.
W.
Hu
,
L.
Lin
,
C.
Yang
, and
J.
Yang
,
J. Chem. Phys.
141
,
214704
(
2014
).
40.
W.
Hu
,
L.
Lin
, and
C.
Yang
,
Phys. Chem. Chem. Phys.
(to be published).
41.
H.
Liu
,
A. T.
Neal
,
Z.
Zhu
,
Z.
Luo
,
X.
Xu
,
D.
Tománek
, and
P. D.
Ye
,
ACS Nano
8
,
4033
(
2014
).
42.
J.
Qiao
,
X.
Kong
,
Z.-X.
Hu
,
F.
Yang
, and
W.
Ji
,
Nat. Commun.
5
,
4475
(
2014
).
43.
L.
Li
,
Y.
Yu
,
G.
Ye
,
Q.
Ge
,
X.
Ou
,
H.
Wu
,
D.
Feng
,
X.
Chen
, and
Y.
Zhang
,
Nat. Nanotech.
9
,
372
(
2014
).
44.
J.
Dai
and
X. C.
Zeng
,
J. Phys. Chem. Lett.
5
,
1289
(
2014
).
45.
W.
Hu
,
T.
Wang
, and
J.
Yang
,
J. Mater. Chem. C
3
,
4756
(
2015
).
46.
K. S.
Novoselov
,
A. K.
Geim
,
S. V.
Morozov
,
D.
Jiang
,
Y.
Zhang
,
S. V.
Dubonos
,
I. V.
Grigorieva
, and
A. A.
Firsov
,
Scinece
306
,
666
(
2004
).
47.
A. K.
Geim
and
K. S.
Novoselov
,
Nat. Mater.
6
,
183
(
2007
).
48.
A. H. C.
Neto
,
F.
Guinea
,
N. M. R.
Peres
,
K. S.
Novoselov
, and
A. K.
Geim
,
Rev. Mod. Phys.
18
,
109
(
2009
).
49.
S.
Goedecker
,
M.
Teter
, and
J.
Hutter
,
Phys. Rev. B
54
,
1703
(
1996
).
50.
C.
Hartwigsen
,
S.
Goedecker
, and
J.
Hutter
,
Phys. Rev. B
58
,
3641
(
1998
).
51.
J. P.
Perdew
,
K.
Burke
, and
M.
Ernzerhof
,
Phys. Rev. Lett.
77
,
3865
(
1996
).
52.
You do not currently have access to this content.