We report the crystal and electronic structures of a noncentrosymmetric quasi-two-dimensional (2D) candidate of topological semimetal AuTe_{2}Br. The Fermi surface of this layered compound consists of 2D-like, topological trivial electron and nontrivial hole pockets, which host a Dirac cone along the *k*_{z} direction. Our transport measurements on the single crystals show highly anisotropic, compensated low-density electrons and holes, both of which exhibit ultrahigh mobility at a level of 10^{5} cm^{2} V^{−1} s^{−1} at low temperature. The highly mobile, compensated carriers lead a nonsaturated, parabolic magnetoresistance as large as 3 × 10^{5} in single-crystalline AuTe_{2}Br in a magnetic field up to 58 T.

## I. INTRODUCTION

The discoveries of Dirac and Weyl semimetal have motivated researchers’ plethoric endeavor for sifting topological materials in crystallographic database.^{1–4} Topological semimetal (TSM) is characterized by topological nontrivial discrete band touching points or nodal lines near which the energy bands display linear crossing in the momentum space, forming Dirac and Weyl cones or nodal rings. The realistic quasielectrons hosted on the cones can introduce many exotic electric properties, including topological nontrivial surface state,^{5–7} extremely high carrier mobility, large magnetoresistance (MR = $\rho H\u2212\rho 0\rho 0$), and chiral anomaly.^{8,9} Till now, we have acknowledged various TSMs, including Na_{3}Bi, Cd_{3}As_{2}, ZrTe_{5}, WTe_{2}, and TaAs family.^{10–14}

Most of the previously reported TSMs show three-dimensional (3D) electronic structures, meaning their Fermi surfaces (FSs) mainly consist of 3D closed pockets in the momentum space. In general, time-reversal invariant 3D topological insulators (TIs) and TSMs are closely related to two-dimensional (2D) quantum spin Hall (QSH) insulators whose edge states are topologically protected.^{15–17} Three dimensional TI and TSM can be built up by stacking the 2D QSH insulating layers along a certain crystal orientation while a QSH effect can be achieved in an exfoliated, monolayer TI and TSM. A well-studied example is the transition metal dichalcogenides (TMDs) whose monolayer atomic crystal can show QSH effect even at high temperature.^{18,19} The bulk electronic structure of TSMs usually degenerates to 3D-like after stacking even though their crystal structures are still highly anisotropic.^{20,21} This electronic structure evolution in dimension is reminiscent to the change from graphene to graphite.^{22}

In this paper, we introduce a layered candidate of TSM, AuTe_{2}Br whose electronic structure persists a 2D-like FS. AuTe_{2}Br belongs to a family of halide AuTe_{2}X (X = Cl, Br, I), which crystalize in a quasi-2D structure consisting of halogen atoms sandwiched by AuTe_{2} networks [Fig. 2(a)].^{23–26} By a comprehensive study of Shubonikov-de Haas (SdH) and de Haas-van Alphen (dHvA) quantum oscillations (QOs) in the single crystal, we found that the Fermi surface (FS) of AuTe_{2}Br mainly consists of separated hole and electron tubes with open orbits along the stacking direction. Moreover, our transport measurements show that the electrons and holes are highly anisotropic while both exhibit ultrahigh mobility at a level of 10^{5} cm^{2} V^{−1} s^{−1} at low temperature. As a result, single-crystalline AuTe_{2}Br manifests nonsaturated, parabolic magnetoresistance as large as 3 × 10^{5} in a magnetic field up to 58 T. Such unique crystal and electronic structures naturally bridge the 3D TSM and 2D electron system, and also provide a platform for the fabrication of the devices based on its topological properties.

## II. EXPERIMENTAL METHOD

We succeeded in synthesis of single crystalline AuTe_{2}Br, following the work of Rabenau *et al.*^{23} 1.4 g gold powder (4N), 0.5 g tellurium powder (4N), and 0.5 ml liquid bromine were sealed in a fused silica ampoule (*ϕ* 10 mm × L 10 cm, T 2.5 mm), and 65% of its capacity was filled with 9M hydrobromic acid. Then, the ampoule was inserted into an autoclave with dry ice to apply balanced pressure outside at high temperature. The autoclave was heated to 350 °C and cooled down to 150 °C uniformly in 10 days. The as-grown samples are soft, silver flakes with a typical dimension of 2.5 mm × 2 mm × 0.2 mm, which are identical to previously reported.

For the single crystal XRD measurements, all of the single crystals from the samples were mounted on the tips of Kapton loop. The data were collected on a Bruker Apex II X-ray diffractometer with Mo radiation *K*_{1}(*λ* = 0.710 73 Å), and the measuring temperature is 100 K. Data were collected over a full sphere of reciprocal space with 0.5° scans in *ω* with an exposure time of 10 s/frame. The 2*θ* range extended from 4° to 75°. The SMART software was used for data acquisition. Intensities were extracted and corrected for Lorentz and polarization effects with the SAINT program. Face-indexed numerical absorption corrections were accomplished with XPREP, which is based on face-indexed absorption.^{27} The twin unit cell was tested. With the SHELXTL package, the crystal structures were solved using direct methods and refined by full-matrix least-squares on *F*^{2}.^{28} Crystallographic data are summarized in Tables I and II.

. | AuTe_{2}Br
. | AuTe_{2}Br
. |
---|---|---|

Formula . | (centrosymmetric) . | (noncentrosymmetric) . |

Temperature (K) | 100 | 100 |

F.W. (g/mol) | 532.08 | 532.08 |

Space group; Z | Cmcm (no. 63); 4 | Cmc2_{1} (no. 36); 4 |

a (Å) | 4.018 (3) | 4.018 (3) |

b (Å) | 12.284 (7) | 12.284 (7) |

c (Å) | 8.885 (5) | 8.885 (5) |

V (Å^{3}) | 438.5 (5) | 438.5 (5) |

Absorption correction | Numerical | Numerical |

Extinction coefficient | 0.000 13 (19) | 0.000 05 (15) |

θ Range (deg) | 3.317–30.428 | 3.317–30.428 |

No. reflections; R_{int} | 1 896; 0.134 2 | 1 896; 0.132 8 |

No. independent reflections | 390 | 595 |

No. parameters | 16 | 27 |

R_{1}; wR_{2} (all I) | 0.094 3; 0.121 7 | 0.092 9; 0.105 6 |

Goodness of fit | 1.201 | 1.131 |

Diffraction peak and hole (e^{−}/Å^{3}) | 4.606; −3.936 | 4.350; −4.355 |

. | AuTe_{2}Br
. | AuTe_{2}Br
. |
---|---|---|

Formula . | (centrosymmetric) . | (noncentrosymmetric) . |

Temperature (K) | 100 | 100 |

F.W. (g/mol) | 532.08 | 532.08 |

Space group; Z | Cmcm (no. 63); 4 | Cmc2_{1} (no. 36); 4 |

a (Å) | 4.018 (3) | 4.018 (3) |

b (Å) | 12.284 (7) | 12.284 (7) |

c (Å) | 8.885 (5) | 8.885 (5) |

V (Å^{3}) | 438.5 (5) | 438.5 (5) |

Absorption correction | Numerical | Numerical |

Extinction coefficient | 0.000 13 (19) | 0.000 05 (15) |

θ Range (deg) | 3.317–30.428 | 3.317–30.428 |

No. reflections; R_{int} | 1 896; 0.134 2 | 1 896; 0.132 8 |

No. independent reflections | 390 | 595 |

No. parameters | 16 | 27 |

R_{1}; wR_{2} (all I) | 0.094 3; 0.121 7 | 0.092 9; 0.105 6 |

Goodness of fit | 1.201 | 1.131 |

Diffraction peak and hole (e^{−}/Å^{3}) | 4.606; −3.936 | 4.350; −4.355 |

Atom . | Wyckoff . | Occupancy . | x . | y . | z . | U_{eq}
. |
---|---|---|---|---|---|---|

Centrosymmetric | ||||||

Au1 | 4c | 1 | 0 | 0.0817 (1) | 1/4 | 0.0066 (5) |

Te2 | 8f | 1 | 1/2 | 0.1060 (1) | 0.0542(2) | 0.0062 (5) |

Br3 | 4c | 1 | 0 | −0.1652 (3) | 1/4 | 0.0090 (8) |

Noncentrosymmetric | ||||||

Au1 | 4a | 1 | 0 | 0.4183 (1) | 0.288(2) | 0.0067 (5) |

Te2 | 4a | 1 | 1/2 | 0.3924 (10) | 0.0920(0) | 0.004 (2) |

Te3 | 4a | 1 | 1/2 | 0.3954 (9) | 0.4835(5) | 0.008 (2) |

Br4 | 4a | 1 | 0 | 0.6652 (4) | 0.284(5) | 0.0088 (13) |

Atom . | Wyckoff . | Occupancy . | x . | y . | z . | U_{eq}
. |
---|---|---|---|---|---|---|

Centrosymmetric | ||||||

Au1 | 4c | 1 | 0 | 0.0817 (1) | 1/4 | 0.0066 (5) |

Te2 | 8f | 1 | 1/2 | 0.1060 (1) | 0.0542(2) | 0.0062 (5) |

Br3 | 4c | 1 | 0 | −0.1652 (3) | 1/4 | 0.0090 (8) |

Noncentrosymmetric | ||||||

Au1 | 4a | 1 | 0 | 0.4183 (1) | 0.288(2) | 0.0067 (5) |

Te2 | 4a | 1 | 1/2 | 0.3924 (10) | 0.0920(0) | 0.004 (2) |

Te3 | 4a | 1 | 1/2 | 0.3954 (9) | 0.4835(5) | 0.008 (2) |

Br4 | 4a | 1 | 0 | 0.6652 (4) | 0.284(5) | 0.0088 (13) |

The resistance and Hall resistance were measured via a standard four-point method, while the magnetic field dependent data in +H and −H were antisymmetrized and symmetrized, respectively, in order to remove the nonsymmetric part due to the contacts. The data in 9 T and 14 T were collected in a Quantum Design Physical Property Measurement System (PPMS-9) and Oxford TeslatronPT (14 T), respectively. Pulsed field measurement up to 58 T was performed at the Wuhan National High Magnetic Field Center. The torque measurement was performed by using a piezoresistive cantilever (SEIKO-PRC400) incorporated into the PPMS-9, on a crystal about 0.1 mm across and 0.01 mm thick of the same batch. A Wheatstone bridge circuit was adopted to detect a small change in the resistance.

## III. RESULTS AND DISCUSSION

Both centrosymmetric and noncentrosymmetric models have been used to refine the crystal structure and our single-crystal X-ray diffraction results revealed that single-crystal AuTe_{2}Br prefers crystallizing in a noncentrosymmetric structure of the space group *Cmc*2_{1} (a = 4.0183 Å, b = 12.284 Å, and c = 8.885 Å), which is different from the previous report.^{26} To further confirm the space group choice, we plot the observed electron distribution contours in both centrosymmetric and noncentrosymmetric models (Fig. 1). The electron density distribution does not match with the centrosymmetric model with slightly off from the expected atomic sites. It is apparent that the observed electron distribution matches well with the noncentrosymmetric model. Comparing with three sibling compounds, the 2D networks of AuTe_{2} ripple in different shapes so that the Br and Cl ion has only one nearest neighboring Au atom whereas the iodine ion has two equally nearest neighboring Au atoms in the upper and down layers, respectively. This structural difference makes the AuTe_{2}Cl and AuTe_{2}Br more 2D-like than AuTe_{2}I, and therefore, their crystals are soft and exfoliable, which promises the fabrication of the electric devices on few layers. Figure 2(b) shows a Scanning Tunnel Microscope (STM) image of an exfoliated crystal of AuTe_{2}Br, which clearly unveils that the b plane consists of the unit cells of 4 Å × 8.9 Å, consistent with our XRD measurement.

Corresponding to its layered crystal structure, AuTe_{2}Br shows a quasi-2D electronic structure in the band dispersion. Our band structure calculation reveals one holelike and one electronlike pocket around the center and edge of the Brillouin Zone (BZ), respectively [Figs. 2(c)–2(e)]. The energy dispersion is much flatter in the (*b*_{1}, *b*_{2}) plane than along the vertical *b*_{3} direction in the reciprocal space, indicating a quasi-2D nature of both electron and hole pockets. N. B. there exists a band-crossing Dirac cone in the hole pocket near the Γ point above the Fermi energy (*E*_{F}) with no spin orbital coupling (SOC). When the SOC is turned on, the high symmetry line opens a very small gap near the crossing point [Fig. 2(f)] while the band structure near the *E*_{F} remains intact.

A single crystal of AuTe_{2}Br shows an uncommonly good metallicity with a room-temperature in-plane resistivity *ρ* ≈ 40 *μ*Ω cm [Fig. 3(a)]. The crystal has a large residual resistance ratio (RRR = $\rho 300K\rho 2K$ = 576) and a small residual resistivity *ρ*_{2K} = 70 nΩ cm in zero magnetic field. Such large RRR and small *ρ*_{2K} are rarely observed in ternary compounds. As comparison, a crystal of Calaverite (AuTe_{2}) shows room-temperature resistivity 270 *μ*Ω cm and RRR = 350.^{29} The *ρ*_{xx} is significantly enhanced under strong magnetic field below 50 K, but at high temperature, it changes slightly. These profiles of temperature-dependent resistivity under different magnetic fields are similar to recently reported semimetals such as PtSn_{4},^{30} WTe_{2},^{31} ZrSiS,^{32} and PtBi_{2}.^{33}

As observed in WTe_{2}, a perfectly compensated semimetal shows a quadratic magnetoresistance (MR) with no sign of saturation in strong field,^{21,31} and that is what we observed for a piece of AuTe_{2}Br in a pulsed magnetic field up to 58 T at 4.2 and 10 K [Fig. 3(b)]: the MR exceeds 3 × 10^{5} at 4.2 K in 58 T with no sign of saturation. The apparent SdH QOs indicate that the carriers move along closed orbits when H∥** b**. In order to clarify the power law of the field dependence at higher temperatures, we measured the same sample’ MR in a superconducting magnetic field up to 9 T. N. B. albeit the MR varies 10

^{10}in 9 T at different temperatures in this double-logarithm scale plot, its power law remains intact as MR ∝

*H*

^{1.9}, very close to the quadratic behavior predicted in a Drude model for a compensated semimetal.

The large MR at 4.2 K for AuTe_{2}Br is comparable to the high-quality WTe_{2} single crystal (RRR > 1000),^{21} which indicates its carriers also have a comparably high mobility. To estimate the mobility at low temperature, we used the relation between MR and magnetic field B = *μ*_{0}H for a compensated semimetal,^{21,31}

A simple scalar mobility $\mu \xaf=\mu h\mu e$ can be estimated as 1.1 × 10^{5} cm^{2} V^{−1} s^{−1} at 4 K.

The densities and motilities of the compensated carriers are estimated by fitting the Hall resistivity *ρ*_{yx} with respect to the field at different temperatures [Fig. 3(c)]. The *ρ*_{yx} is positively, linearly dependent on the field above 100 K, leading to hole carrier concentration *n*_{h} = 1.45 × 10^{20} cm^{−3} by a single band model. Below 50 K, the *ρ*_{yx} shows nonlinear dependence, concurrently with the large parabolic MR. A two-band model fitting below 50 K leads to *n*_{h}(*n*_{e}), denoting the carrier concentrations of holes (electrons), approximate to each other at a level of 10^{20} cm^{−3}. The *μ*_{h} and *μ*_{e}, denoting the mobility of holes and electrons, are estimated as 3.5 × 10^{5} cm^{2} V^{−1} s^{−1} and 0.7 × 10^{5} cm^{2} V^{−1} s^{−1} at 2 K, respectively. These results are further confirmed by the two-band model fitting of the Hall conductivity ($\sigma xy=\rho yx\rho xx2+\rho yx2$) in which we use the constrains zero-field resistivity [*ρ* = *e*(*μ*_{h}*n*_{h} + *μ*_{e}*n*_{e})] and *MR* = *μ*_{h}*μ*_{e}*B*^{2} [Fig. 3(d)]. As shown in Fig. 3(e), the fitting results are close to the fitting of *ρ*_{yx}.

The existence of ultrahighly mobile carriers in AuTe_{2}Br can be confirmed by Drude model in which the conductivity in zero field (*σ* = *ρ*^{−1}) equals *e*(*μ*_{h}*n*_{h} + *μ*_{e}*n*_{e}). Given *n*_{h} ≈ *n*_{e} ≈ 10^{20} cm^{−3}, we estimate that $\mu h+\mu e2\u22485\xd7105cm2\u2009V\u22121\u2009s\u22121$ at 2 K. The three different methods give us a reliable estimation of highly mobile carriers in AuTe_{2}Br.

Given that the MR of AuTe_{2}Br is comparable to other high-mobility TSMs in magnitude, it is much more anisotropic than any TSMs reported before. Figure 3(d) shows extremely large anisotropic transversal angular-dependent MR whose profile is proportional to cos^{2} *θ*, indicating the dominant rule of the *μ*_{0}H along the stacking ** b** direction. If we assume the transversal conductivity

*σ*(

*θ*) ≈

*ρ*

^{−1}(

*θ*) equals the sum of conductivity of 2D and 3D electrons [

*σ*

_{2D}(

*θ*) and

*σ*

_{3D}(

*θ*)], then

Assuming the transversal *σ*_{3D} is independent on *θ*, then the ratio of *ρ*(*θ* = 0°) to *ρ*(*θ* = 90°) gives the value of $\sigma 2D\sigma 3D$. This ratio is more than 100 under the magnetic fields higher than 3 T, and we infer that *σ*_{2D} contributes more than 99% of the total in-plane conductivity. As comparison, the ratios of $\rho (\theta =0\xb0)\rho (\theta =90\xb0)$ is approximately 2–3 for previously reported quasi-2D TSMs AMnSb_{2} and AMnBi_{2} (A = alkaline earth)^{34–37} and LaAgBi_{2}.^{38} Further investigation of angular-dependent MR is needed to elucidate the angular dependent oscillations and the sharp peak at 90° and 270° (see the caption of Fig. 3).

We now extract more information about the 2D and 3D carriers in AuTe_{2}Br from its field dependent QOs. As shown in Fig. 4(a), the 2D hole pocket (named as *β*) is a gourdlike tube with peanut-shape maximum and minimum cross sections, while the 2D electron pocket (named as *γ*) is a corrugated, serpentine tube with nearly circular shape of cross section. The projected extremal orbits of *β* and *γ* and the small raisinlike electron ellipsoids (named as *α*) along the ** b** direction are shown in Fig. 4(b).

The magnetic torque signal and electrical resistivity show significant QOs starting at 3 T at low temperatures when the field is along ** b** direction [Fig. 4(c)]. The oscillatory part of the torque and the MR with respect to reciprocal of the field ($1\mu 0H$) is shown in Fig. 4(d). Fast Fourier transform (FFT) discerns five base frequencies in SdH QOs, which we named as

*F*

_{α},

*F*

_{β1},

*F*

_{β2},

*F*

_{γ1}, and

*F*

_{γ2}and their multiplications [Fig. 4(e)]. Our result is apparently different from previous report, which shows four frequencies (52, 78, 105, and 173 T) in the SdH QOs.

^{41}Thanks to the simple topology of the FS, all the distinct frequencies can be identified completely corresponding to the extremal orbits of electron and hole pockets. The strong QOs of

*F*

_{β1}and

*F*

_{β2}stem from the peanut-shape cross sections of topological hole tube

*β*, while the low frequency

*F*

_{α}stems from the tiny 3D electron pocket

*α*. The two close weak QO frequencies

*F*

_{γ1}and

*F*

_{γ2}originate from the serpentine trivial electron tube

*γ*whose cross section along

**is nearly uniform. The resistivity in pulsed fields up to 58 T reveals an identical FFT spectrum as that in low fields [Fig. 4(e)]. As comparison, the mapped Fermi surface in previous work seems to be inconsistent with the band structure calculation.**

*b*^{41}

Identifying all the cross sections of the electron and hole pockets means we can address the carriers properties completely by analyzing their QOs. The parameters of the cyclotron masses (*m*^{*}) for different frequencies are obtained by using Lifshitz-Kosevich (LK) formula,^{42}

where $\beta =e\u210fB/m*$, *k*_{B} is Boltzmann constant, *e* is electron charge, and $\u210f$ is Planck constant over 2*π*. We fit the temperature dependent FFT amplitudes of the dHvA and SdH QOs for each frequency in Fig. 4(f), and the results are shown in Table III. The cyclotron masses for each pocket are in a short range of 0.07–0.1*m*_{e}. Then, we calculated the cross section $AF=2\pi e\u210fF$, the Fermi vector $kF=AF\pi $, and the Fermi velocity $vF=\u210fkFm*$ for each orbital. Note the maximum and minimum cross sections of *β* tube are very different, and therefore, its parameters cannot be estimated precisely. The Fermi energy $EF=m*vF2$ is estimated for each pocket, and we found the values are consistent with the calculation (Fig. 2). The nearly identical parameters for *F*_{γ1} and *F*_{γ2} confirm their same origination from *γ* tube.

Cross section . | F_{α}
. | F_{β1}
. | F_{β2}
. | F_{γ1}
. | F_{γ2}
. |
---|---|---|---|---|---|

F-measured (T) | 18.3 | 82.2 | 179.5 | 109.5 | 115.6 |

F-calculated (T) | 19 | 91 | 179 | 134 | |

m^{*} (m_{e}) | 0.097 | 0.074 (0.072) | 0.081 (0.081) | 0.084 | 0.084 |

k_{F} (Å^{−1}) | 0.024 | 0.050 | 0.074 | 0.058 | 0.059 |

v_{F} (10^{5} m/s) | 0.28 | 0.78 | 1.05 | 0.79 | 0.82 |

E_{F} (eV) | 0.04 | 0.26–0.51 | 0.30–0.32 | ||

n/p (10^{19} cm^{−3}) | 0.06 | 5.2 | 4.6 |

Cross section . | F_{α}
. | F_{β1}
. | F_{β2}
. | F_{γ1}
. | F_{γ2}
. |
---|---|---|---|---|---|

F-measured (T) | 18.3 | 82.2 | 179.5 | 109.5 | 115.6 |

F-calculated (T) | 19 | 91 | 179 | 134 | |

m^{*} (m_{e}) | 0.097 | 0.074 (0.072) | 0.081 (0.081) | 0.084 | 0.084 |

k_{F} (Å^{−1}) | 0.024 | 0.050 | 0.074 | 0.058 | 0.059 |

v_{F} (10^{5} m/s) | 0.28 | 0.78 | 1.05 | 0.79 | 0.82 |

E_{F} (eV) | 0.04 | 0.26–0.51 | 0.30–0.32 | ||

n/p (10^{19} cm^{−3}) | 0.06 | 5.2 | 4.6 |

^{a}

The calculated extremal cross section of *γ* tube cannot distinguish *F*_{γ1} and *F*_{γ2}.

In order to estimate the carrier densities of hole and electron tubes, we treat their FSs as cylinders with a height of $kb=2\pi b$ and then use the relationship $n=AFkb4\pi 3$ to get *n*_{h−2D} = 5.17 × 10^{19} cm^{−3} and *n*_{e−2D} = 4.56 × 10^{19} cm^{−3}. This result is close to the two-band model fitting. The carrier density of the small 3D electron pocket was estimated as $ne\u22123D=kF33\pi 3\u22480.06\xd71019cm\u22123$, about 1% of the 2D carriers. All above confirm AuTe_{2}Br as a compensated quasi-2D semimetal.

We can extract more information about the strongest QOs of *F*_{β1}, which corresponds to the minimal cross section of the topological hole pocket *β*. The quantum life time $\tau Q=\u210f2\pi kBTD=6.3\xd710\u221214\u2009s$ is calculated by fitting the Dingle temperature *T*_{D} = 19.3 K in Fig. 4(g) at 2 K. Given *μ*_{h} ≈ 3 × 10^{5} cm^{2} V^{−1} s^{−1}, the transport lifetime $\tau tr=\mu hm*e$ is estimated as 1.3 × 10^{−11} s, leading to a ratio of $\tau tr\tau Q=200$. The large ratio of transport lifetime and quantum lifetime is commonly observed in various TSMs.^{15,43} It is well known that *τ*_{tr} measures backscattering processes that relax the current while *τ*_{Q} is sensitive to all processes that broaden the Landau sub-bands (LBs). The large ratio indicates that the small-angle scatterings are dominant while the backscattering is strongly protected at low temperature. We notice that the SdH QO pattern is complicated because *F*_{β2} is close to the double frequency of *F*_{β1}. Therefore, it is difficult to extract the accurate phase information based our measurements.

Figure 5 compares the SdH frequencies for the magnetic field along different directions with the calculation. The frequencies *F*_{β1} and *F*_{β2} approximately follow the $1cos\u2009\theta $ and $1cos\u2009\phi $ relations at low tiled angles. The extremal cross section for the serpentine tube *γ* changes in a complicated manner in high tilted angles in which the electrons can have several close extremal orbits. In experiment, we do not observe any discernable frequencies from those orbitals when the tilted angle is higher than 60°. In contrast, the small electron ellipsoid changes in a different manner.

The quasi-2D electronic structure of AuTe_{2}Br is closely related with its unique crystal structure. As shown in Fig. 2(a), the AuTe_{2} layer is distinct from the TMDs in which the transition metal has six coordinated chalcogen atoms. Here, every Au atom has planar four-coordinated Te atoms, which connect with other Te atoms in isolated Te–Te dimers (2.94 Å). The Te–Te dimers and Au atoms weave a ripplelike 2D network. This family of compounds has 18*e* in total if we account that the Au, Te, and halogen atoms each contribute 13*e*, 2*e*, and 1*e*, respectively. However, we notice that the Te–Te dimer should share one pair of electrons, and therefore, it should follow 16*e* rule instead of 18. The 16*e* configuration is commonly observed in many transition metal square planar complexes containing Ir(i), Pd(ii), or Au(iii), in which the transition metal’s $dx2\u2212y2$ orbital has significantly higher energy than the rest of the d orbitals.^{44} The molecular orbital (MO) *b*_{1g} of $dx2\u2212y2$ orbital parentage is the lowest unoccupied for 16*e* count, which leads to a gap or pseudogap between the MOs of the rest d orbitals parentage.

It is noteworthy that AuTe_{2} can form different types of the layers in other quasi-2D layer compounds. For example, in the charge balanced $[Pb2BiS3]+[AuTe2]\u2212$, the $[AuTe2]\u2212$ layer apparently obeys the 18*e* rule with no Te–Te dimers.^{45} Previous study revealed a 2D compensated semimetal, which hosts highly mobile carriers as well. Future study of the devices based on the exfoliated two types of AuTe_{2} layers will be interesting.

In summary, we grew large single-crystalline AuTe_{2}Br and accomplished resistance and torque measurements in magnetic field. Our measurements reveal that it is a quasi-2D, compensated TSM hosting highly mobile electrons and holes. Its unique crystal and electronic structure highlights a novel 2D topological electron system whose physical properties should be investigated in depth in the future.

## ACKNOWLEDGMENTS

The authors thank Chenglong Zhang, Gang Xu, Haizhou Lu, Jinglei Zhang, and Qiang Gu for valuable discussions. This work was supported by the National Natural Science Foundation of China (Grant Nos. U1832214 and 11774007), the National Key R&D Program of China (Grant No. 2018YFA0305601), and the Key Research Program of the Chinese Academy of Science (Grant No. XDPB08-1). S.J. was supported by Jiangsu Province Program for Entrepreneurial and Innovative Talents, Kunshan Program for Entrepreneurial and Innovative Talents and Natural Science Foundation of Jiangsu Province of China (Grant No. BK2016). T.-R.C. was supported by the Ministry of Science and Technology under MOST Young Scholar Fellowship: the MOST Grant for the Columbus Program No. 107-2636-M-006-004, National Cheng Kung University, Taiwan, and National Center for Theoretical Sciences (NCTS), Taiwan. The work at LSU was supported by Beckman Young Investigator (BYI) program.