Structural, vibrational, and electronic topological transitions of Bi_1.5Sb_0.5Te_1.8Se_1.2 under pressure

Topological insulators (TIs) have attracted research interest in the last several years, especially due to their conducting edge states (in 2D TIs) or surface states (in 3D TIs) that are topologically protected against backscattering and cannot be passivated or destroyed by impurities or imperfections.^1–4 This leads to “dissipationless” transport by the edge or surface states that could be realized in a variety of applications including spintronics and quantum computing, and possibly create the elusive “Majorana fermion.”⁵ Among the many TIs that have been identified, the metal chalcogenides of the A₂B₃ (A = Bi, Sb; B = S, Se, Te) series and their solid solutions have been studied extensively from the perspective of topological behavior as well as thermoelectric properties.^6–10 Specifically, the composition of Bi_1.5Sb_0.5Te_3-ySe_y with y = ∼1.2 was found to have the highest bulk resistivity, making the compound optimal for studying the surface transport behaviors.^6,7,9

The TI states of the A₂B₃ chalcogenide family are likely closely related to their crystal structures. Reportedly, R$3¯$m crystals such as Bi₂Te₃, Sb₂Te₃, and Bi₂Se₃ are TIs, whereas Pnma crystals such as Sb₂Se₃, Bi₂S₃, and Sb₂S₃ are topologically trivial in ambient conditions.¹¹ Therefore, it is of great importance to understand the role of crystal structure in the topological behavior. An effective method for probing this relationship is via the application of hydrostatic pressure, which is a powerful means to modulate lattice parameters and crystal structures of a material, as well as its electronic structures and topological states.^12,13 Pressure-induced crystal structural transitions in the R$3¯$m structured chalcogenide family have been well explored, and their electronic topological transitions (ETTs) within the TI phase are also reported.^14–20 On the contrary, Pnma-structured chalcogenides remain structurally stable up to 25 GPa, and their pressure-induced ETTs are under debate.^21–26 It is notable that a pressure-induced TI state in Sb₂Se₃ has been reported, despite some outstanding debates relating to the pressure transmitting medium (PTM).^22,27,28 Importantly, the relationship between crystal phase transitions and topological transitions (e.g., whether the surface TI state is exclusively dependent, or irrelevant to the crystal structure) remains to be clarified.^22,29,30

In this paper, we report a combined theoretical and experimental investigation of the effect of hydrostatic pressure on the topological insulator quaternary alloy Bi_1.5Sb_0.5Te_1.8Se_1.2. Two structural phase transitions were observed from X-ray diffraction (XRD) and Raman spectroscopy and confirmed by first-principles calculations: from the ambient pressure rhombohedral R$3¯$m phase to a monoclinic C2/m phase at ∼13 GPa, and to a tetragonal disordered I4/mmm phase at ∼22 GPa. Furthermore, within the ambient R$3¯$m phase, a series of electronic transitions were demonstrated by the evolution of the band structure as a function of pressure. The bulk indirect band gap of the quaternary alloy undergoes indirect-to-direct (I-to-D) band gap transition at ∼5.8 GPa. At ∼8.24 GPa, the bulk band gap closes and develops a linear crossing of bands to form a 3D Dirac semimetal (DSM) state. On further increase in pressure, the system becomes a normal metal before it undergoes the first structural phase transition to the metallic C2/m and I4/mmm phases. Concomitant with the pressure-induced electronic transitions, various observations from experiments also revealed profound changes at these transitions, such as c/a ratio anomaly from XRD, and pressure coefficient change, and full width at half maximum (FWHM) anomalies from Raman spectroscopy. This result implies that, although closely related, the surface states of TIs could be modulated without change in crystal structure under hydrostatic pressures. Also, in comparison to binary end member compounds, the Bi_1.5Sb_0.5Te_1.8Se_1.2 alloy showed structural phase transition and other anomalies at higher pressures. Change in axis compressibility and increase in electron-phonon coupling (EPC) are suggested to be precursors of the structural transitions.

Bi_1.5Sb_0.5Te_1.8Se_1.2 single crystals were grown in a two-step process. The material was first synthesized by a modified vertical Bridgman technique similar to that reported by Wang et al.⁹ Stoichiometric quantities of high purity (≥5 N) metals of bismuth (5 N), antimony (5 N), selenium (6 N) and tellurium (7 N) were vacuum sealed (10⁻⁶Torr) in a fused silica ampoule and melted at 900 °C for two days and slowly cooled to room temperature. The reacted material was then resealed under vacuum in a growth ampoule with a taper. The ampoule was heated to 900 °C for two days, cooled down to 550 °C over 180 h, and held at that temperature for 4 days before being quenched in liquid nitrogen and brought to room temperature. The resulting crystal was highly crystalline and could be cleaved or exfoliated using the Scotch tape method.

A symmetric diamond anvil cell with a pair of 350-μm diameter diamond culets was used for the high-pressure experiments. Rhenium gaskets pre-indented to ∼40 μm in thickness were drilled at the center to form the sample chamber with a diameter of ∼200 μm. A piece of Bi_1.5Sb_0.5Te_1.8Se_1.2 was cleaved and then placed near the center of the sample chamber, along with a ruby sphere used as the pressure indicator. The shift of ruby R₁ line was used to calibrate the pressure in situ with uncertainty of <1%.^31–33 Neon pressure medium was loaded at the Mineral Physics Lab at the University of Texas at Austin. The thermal conductivity of Ne PTM at the pressure range used in this work is above 0.3 W/m K, which is larger than the thermal conductivity of methanol–ethanol mixture PTM at ambient pressure.^34,35 Raman and ruby spectra were collected using a Renishaw inVia Raman spectroscopy system equipped with a 532-nm green laser and a 2400-line/mm grating. To prevent thermal damage, laser power was restricted to below 3 mW. The spectral resolution of the Raman spectroscopy is 1.2 cm⁻¹, and the spectrometer was calibrated for each measurement within ±0.5 cm⁻¹ error range using Si reference sample. Samples for X-ray diffraction (XRD) experiments were ground to form a randomly oriented polycrystalline powder. An Au particle and ruby were also loaded as pressure indicators. The XRD experiments were conducted at GSECARS 13IDD beamline of the Advanced Photon Source, Argonne National Laboratory. X-ray diffraction patterns were collected at various pressures using an X-ray beam of wavelength 0.3344 Å and spot size of 2–3 μm. Images of powder diffraction patterns were recorded using a MAR345 CCD detector with pixel size 79 × 79 μm². Obtained images were integrated using FIT2D.³⁶ XRD patterns were analyzed by Le Bail refinement method using FullProf software.³⁷ Backgrounds of the XRD data were modeled by selecting points manually to create a smoothly varying background profile. Peak profiles of the XRD data were modeled using Thompson-Cox-Hastings pseudo-Voigt functions. For all XRD patterns, unit cell parameters and half-width parameters were refined.

Theoretical calculations were performed using first-principles density functional theory (DFT)³⁸ as implemented in the Vienna Ab initio Simulation Package (VASP).^39,40 Projector augmented wave (PAW)^41,42 pseudopotentials were used to represent the ion–electron interactions. The Perdew-Burke-Ernzerhof (PBE)⁴³ generalized gradient approximation was used to approximate the exchange and correlation part of the total energy. The kinetic energy cut-off for a plane wave basis was set to 450 eV. All electronic structure calculations were carried out on fully optimized special quasirandom structures (SQS), generated using the Monte Carlo SQS tool as implemented in Alloy Theoretical Automated Toolkit (ATAT).⁴⁴ For feasible computations, the structures were modeled at the composition Bi_1.5Sb_0.5Te_1.5Se_1.5, which marginally differs from the experimental composition Bi_1.5Sb_0.5Te_1.8Se_1.2. All the structures were fully relaxed by employing a conjugate gradient scheme until the Hellmann-Feynman forces on the atoms were less than 0.005 eV/Å. The topological invariant quantity Z₂ was calculated by the method of evolution of Wannier charge centers (WCC), as implemented in the Z2Pack.⁴⁵ The evolution of WCC along k_y, for example, corresponds to the phase factor θ of the eigenvalues of the position operator projected onto the occupied subspace.^45–47 The topological invariant Z₂ was then calculated by counting the number of crossings of any random horizontal reference line with the evolution of θ's, modulo two. The Bloch spectral function in the computed angle-resolved photoemission spectroscopy (ARPES) and surface projected band structure were calculated based on the iterative Green's function method⁴⁸ based on the tight-binding Hamiltonian from the maximally localized Wannier functions (MLWF),⁴⁷ as implemented in the WannierTools package.⁴⁹ 

X-ray diffraction patterns of Bi_1.5Sb_0.5Te_1.8Se_1.2 were taken up to 45 GPa in order to study the structural behavior of the quaternary alloy. Figure 1 shows the selected XRD patterns, and all XRD patterns collected up to 45 GPa are provided in supplementary material Fig. S1. The Bi_1.5Sb_0.5Te_1.8Se_1.2 crystallizes in rhombohedral structure with space group R$3¯$m at ambient pressure [Phase I; Figs. 2(a) and S2 (supplementary material)]. With increasing pressure, the first structural phase transition was observed in the pressure range of 13–15 GPa, where new X-ray diffraction peaks appeared and co-existed with Phase I. Observed above the transition was a monoclinic phase (Phase II) with space group C2/m [Figs. 2(b) and S3 (supplementary material)]. Further increase in pressure resulted in an additional transition to a tetragonal I4/mmm phase [Phase III; Figs. 2(c) and S4 (supplementary material)] at 22–24 GPa, which persisted up to the highest pressure carried out in the present study. This structural sequence observed is similar with parent compound Bi₂Se₃.^16,50 Unit cell volume per formula unit for each of the three phases is fitted using a third order Birch-Murnaghan equation of state (EoS) in Fig. 3(a).^51,52 Volume collapse of 3.52% and 2.32% was observed across the first and second phase transitions, respectively. Bulk modulus at ambient conditions (K₀), first derivative of the bulk modulus at ambient conditions (K₀′), and zero pressure volume per formula unit (V₀/Z) for the three phases are given in Table I. K₀ values obtained for Bi_1.5Sb_0.5Te_1.8Se_1.2 in phase R$3¯$m, C2/m, and I4/mmm were 52(2), 61(2), and 66(6) GPa, respectively. For phase R$3¯$m and C2/m, K₀ values are comparable to reported bulk moduli of parent compound Bi₂Se₃ (48–53 GPa, and 60–64 GPa, respectively), despite the composition closer to Bi₂Te₃.^16,50 In the I4/mmm phase, however, K₀ for Bi_1.5Sb_0.5Te_1.8Se_1.2 is observed to be much lower compared to Bi₂Se₃. It is suggested that alloying with Sb₂Te₃ does not alter the structure but dramatically softens the I4/mmm phase. It is notable that a few weak peaks coexisting with phases C2/m and I4/mmm in the pressure range of 16 to 30 GPa were observed. These peaks could presumably be attributed to the C2/c phase, but they are too weak to draw a definitive conclusion.

Figures 3(b) and 3(c) show the normalized lattice parameters and axial ratio c/a as a function of pressure for the ambient R$3¯$m phase. We have also fitted pressure-dependent a and c lattice parameters using a modified Birch-Murnaghan EoS, as shown in Fig. S5 (supplementary material).^16,50 The K_a,0 and K_c,0 values were 56(4) and 47(2) GPa, respectively. Due to the anisotropy of compressibility between intra-quintuple layers (QL) and van der Waals type inter-QL, the axial c parameter is more compressible than the axial a parameter.¹² However, c/a ratio showed an anomalous minimum at ∼7 GPa. Anomalous minimum followed by an increase in the c/a ratio is frequently regarded as indicative of ETT or Lifshitz transition, despite existing debates on the origin.^{16,29,53–56} Polian et al.⁵⁵ reported for Bi₂Te₃ that the c/a anomaly is due more to the pressure variation, or EoS anomaly, of the in-plane lattice parameter rather than the out-of-plane lattice parameter. In Bi_1.5Sb_0.5Te_1.8Se_1.2, the c/a anomaly was observed at a higher pressure (∼7 GPa) that is comparable to the value of ∼8.2 GPa at which the theoretical calculations predict an isostructural ETT from a topological insulator to a three-dimensional DSM. Raman spectra as a function of pressure, as discussed latter, also revealed significant changes in the linewidths at ∼7 GPa. Based on these observations, it is surmised that the c/a anomaly in the quaternary alloy is also related to an isostructural transition of topological origin.

To confirm the experimentally observed phase transitions as a function of pressure, different phases were modeled using the special quasirandom algorithm (SQS),^57,58 which is a very useful modeling approach especially for the case of disordered alloys. These SQSs mimic the relevant local atomic structure of the random alloy, thereby providing a complete representation of its electronic structure.⁵⁷ In the present work, the SQSs are generated for three different phases of the quaternary alloy, namely R$3¯$m, C2/m, and I4/mmm. The choice of the size of the supercell to generate the periodic structure (having integer number of each element after generating the supercell) depends on the concentration of each element in the alloy. To handle the system computationally, the minimum possible supercell size of 2 × 2 × 2 is taken for all the three phases with the alloy concentration of Bi_1.5Sb_0.5Te_1.5Se_1.5, instead of Bi_1.5Sb_0.5Te_1.8Se_1.2.

The primitive unit cells of the SQSs in the three phases R$3¯$m, C2/m, and I4/mmm and the corresponding Brillouin zones are shown in Figs. 2(a)–2(c). The total number of atoms in the unit cell are 40, 80, and 40 for R$3¯$m, C2/m, and I4/mmm, respectively, where in each of these phases, 25% of the total Bi sites are replaced by Sb atoms and 50% of the Te sites are replaced by Se atoms in the primitive unit cells of Bi₂Te₃ using quasi-random approach. These structures are then relaxed, and the optimized parameters are a = b = c = 20.7 Å; a = 8.4 Å, b = 16.9 Å, c = 20.1 Å; and a = b = c = 20.2 Å for R$3¯$m, C2/m, and I4/mmm, respectively. Since we have used a primitive cell for the DFT calculations, we can only compare the bulk modulus, its pressure derivative, and the volume per formula unit for these phases, instead of lattice parameters. These values are summarized in Table II. The slight difference from the experimentally reported values (Table I) may be due to the difference in the concentration of modeled system from the experimental one. Further, the phase transitions between these phases are determined by calculating the energy/f.u. as a function of the volume/f.u., as well as the enthalpy difference relative to R$3¯$m phase as shown in Figs. 4(a) and 4(b). The ambient pressure phase R$3¯$m undergoes a phase transition at 13.7 GPa to C2/m phase. On further increase in pressure, a second phase transition to I4/mmm phase occurs at 24.7 GPa. The transition pressures match closely with the experimental results.

To investigate the electronic and topological properties, electronic band dispersions were calculated. The electronic band structures for the R$3¯$m phase with and without spin-orbit coupling (SOC) are shown in Figs. 5(a) and 5(b), and the band structures for the two high pressure phases are shown in Fig. S7 (supplementary material). The ambient pressure phase is a direct band gap semiconductor without SOC. However, due to the presence of heavy elements, the dispersion of states both in the valence and conduction bands are affected by the inclusion of SOC, causing the phase to become an indirect band gap semiconductor. Interestingly, the orbital contribution also changes its order across the Fermi level around the Γ point when SOC is applied. In the absence of SOC, the valence band maxima (VBM) at the Γ point has major contributions from Te-p_x and Se-p_x orbitals, and minor contribution from Bi-s orbitals (maximum contribution from Bi-p_x orbitals), whereas the conduction band minima (CBM) has large contribution from the Bi-p_x orbital, as shown in the inset of Fig. 5(a). When SOC is applied, the valence bands around the Γ point acquire major contribution from Bi-p_x orbital and the conduction bands from Te-p_x and Se-p_x orbitals, as presented in the inset of Fig. 5(b). This indicates the inversion in the band character between the p_x-orbitals of the atoms in the alloy with the inclusion of SOC. Hence, the quaternary alloy in its parent phase is a topological insulator even after alloying Bi₂Te₃ with Se and Sb. The non-trivial topology in the parent phase is further confirmed by topological invariant Z₂ through the evolution of Wannier charge centers. The Z₂ for this phase is 1, and thus has non-trivial topology.

To benchmark the reliability of our predictions, we further calculated Z-terminated (001) Bloch spectral functions with SOC for bulk as well as for both bulk and surface, based on the idea of the bulk-edge correspondence of topological insulators.⁴¹ The tight-binding model, based on maximally localized Wannier functions, reproduces the DFT band structure, as shown in Fig. S6 (supplementary material) and simulates the ARPES with the calculated surface density of states. The computed ARPES in Fig. 5(d) clearly shows the Dirac cone-like surface states, thereby confirming the non-trivial topology.

Having determined the ambient pressure phase to be a topological insulator and the higher pressure phases as normal metals, we are thus interested in determining the evolution of the band structure and the electronic topology of the R$3¯$m phase before it undergoes its first phase transition. Hence, the band structures in the R$3¯$m phase at various pressures are examined, and selected band structures at particular pressures are shown in Fig. 6. As discussed previously, at zero pressure, the quaternary alloy is a topological semiconductor, with indirect bulk band gap having VBM at Z|N and CBM at M|R. With an increase in pressure, the conduction bands start to move down along Z|N direction, whereas the CBM along M|R direction starts to move up slightly. At 5.80 GPa, the alloy becomes a direct band gap semiconductor, with both VBM and CBM at the Z|N high symmetry point. Upon further increase in the pressure, the dispersion along the Z|N–Γ high symmetry direction comes closer to the Fermi-level, whereas the remaining states are gapped along the other high symmetry directions. At 8.24 GPa, the band gap along the Z|N–Γ direction vanishes completely, creating a linear touching of bands. The ambient topological phase of quaternary alloy thus undergoes a transition to a non-trivial Dirac semimetal (DSM) state. Accordingly, the surface state will mix with bulk state and lose its topologically protected state as the bulk band gap collapses.^11,59 On increasing the pressure above 8.24 GPa, the states causing the linear dispersion start to move up above the Fermi level, as the states along other high symmetry directions move closer to the Fermi-level. At 12.09 GPa, which is below the first phase transition pressure to the C2/m phase, the system becomes a normal metal. The band dispersions for the high-pressure C2/m and I4/mmm phases show topologically trivial and metallic behavior (Fig. S7, supplementary material). It is notable that since DFT has a tendency to underestimate the band gap, and the modeled system has slightly different concentration than the experimental one, the theoretical values of the band gap, bulk modulus as well as the transition pressures will be slightly different from those observed experimentally.

Raman spectra were measured as a function of pressure up to 30.7 GPa, and selected spectra are shown in Fig. 7(a). The phase transitions were observed with good agreement with XRD and theoretical predictions. At 10.1 GPa, two new peaks denoted as P1 and P2 appeared at 25–50 cm⁻¹, clearly indicating the first phase transition. Peak P3, which is relatively isolated and well-defined, also showed abrupt changes in its FWHM and intensity with the transition (Fig. S8, supplementary material). At 24.1 GPa, all the features in the Raman spectra vanished, confirming the second phase transition to the disordered I4/mmm phase. It should be noted that Raman spectroscopy probes the material close to the surface, in contrast to XRD, and it is therefore likely that the phonon spectra from the surface layer are more affected at the onset of the transition.

Listed in Table III are the observed Raman peak positions derived from Lorentzian fitting of the line shapes. The R$3¯$m phase has two transverse (E_g) and two longitudinal (A_1g) Raman-active zone-center phonon modes, and the C2/m phase has 15 Raman active modes (10A_g + 5B_g).^{16,17,19,59,60} Alloying can induce continuous shift of the Raman frequencies with the composition (one-mode behavior), discontinuities and peak splitting for intermediate value of composition (two-mode behavior), or disorder-related modes.^13,61,62 The $Eg1$ and $A1g1$ modes exhibit one-mode behavior upon metal or chalcogen alloying, whereas $Eg2$ and $A1g2$ modes exhibit two-mode behavior upon chalcogen alloying.^61,63 In this regard, the frequencies and number of Raman peaks of Bi_1.5Sb_0.5Te_1.8Se_1.2 are not necessarily identical to those of binary end members. For example, P3 could be assigned as $A1g1$ mode based on the proximity of the frequency. However, it is unlikely a mode from a specific binary compound, but likely originated from a continuous frequency shift with respect to the compositional alloying. On the other hand, P4-7 are results of both one-mode and two-mode behavior. By comparing the frequencies to the binary end members and close component BiSbTe₂Se, it is suggested that P4-5 and P6-7 have originated from $Eg2$ and $A1g2$ modes, respectively.^61,64 The large (small) pressure coefficient of P4 (P7), which will be discussed shortly, also confirms the origin of the Raman modes.^16–19,29 P1 and P2 peaks in C2/m phase are suggested to be $Ag1$ or $Bg1$ mode, based on the frequencies and their negligible pressure dependence.^16,17,19 Further theoretical and experimental work is suggested to identify the origin of the two peaks. The highest energy mode (P8) at ∼225 cm⁻¹ is likely a chalcogen A₁ mode, which is a sign of crystallized chalcogen defects.^65–67 Note that due to the notably weak intensity and high frequency, the influence of P8 on the analysis of P1-7 is marginal. Also, P8 does not show a monotonic increase in its intensity (inset of Fig. S9, supplementary material) despite repeated Raman measurements on the same spot, which implies that the laser beam does not induce ongoing damage to the sample.

Figure 7(b) shows the positions of the Raman peaks P1-P8 as a function of pressure. Peaks P3-P7 hardened with increasing pressure, due to the decreasing bond lengths under pressure. The peak positions of selected peaks up to ∼9 GPa are enlarged in Figs. 7(c) and 7(d). P8 showed clear redshift with a pressure coefficient of −2.88 cm⁻¹GPa⁻¹ [Fig. 7(d)], in good agreement with previous reports.^65–67 From linear fits of regions below and above 4 GPa, the pressure coefficients of P3 and P4 exhibit clear decrease, whereas those of P7 and P8 show very little change. The changes in pressure coefficients are also observed for the $A1g1$ and $Eg2$ modes of binary Bi₂Te₃,¹⁷ Bi₂Se₃,¹⁶ and Sb₂Te₃,¹⁹ speculated to be an indication of an isostructural ETT or Lifshitz transition. However, our DFT calculations do not reveal any anomalous topological behavior near ∼4 GPa, other than the indirect-to-direct (I-to-D) band gap transition at ∼5.80 GPa (Fig. 6). Bera et al. related the pressure coefficient change in Bi₂Se₃ to an isostructural transition featuring rapid decrease in c/a ratio and increase in internal bond angle α.²⁹ However, such changes in a-axis and c-axis compressibility could be related to the rapid and heterogeneous changes in band structures, such as rapid lowering of VB, particularly at Z|N direction. Further studies are warranted to identify the origin of the pressure coefficient change of the Raman modes in Bi_1.5Sb_0.5Te_1.8Se_1.2. It is notable that the Ne pressure medium crystallizes at 4.8 GPa, but non-hydrostaticity starts to develop only at higher pressures of 15–20 GPa.^33,68 Also, considering the stable FWHM of P3, the crystallization of Ne medium is unlikely to cause the pressure coefficient change. Thermal shifting of the Raman peaks could be also ruled out since the $Eg2$ and $A1g2$ modes in Bi₂Te₃ family have similar temperature-dependent shifts.⁶⁹ 

The FWHMs of the peaks P4-P7 from Bi_1.5Sb_0.5Te_1.5Se_1.5 are shown in Fig. 8, with linear fits. The FWHMs of P4 and P5 peaks were suppressed up to ∼7 GPa and increased substantially in the pressure range 7–11 GPa. The FWHMs reached maxima at ∼11.5 GPa, followed by rapid decreases. Considering that the increase in FWHMs is rather gradual compared to the abrupt changes in P1-P3 with phase transition, and the fact that the FWHMs decrease after the maxima, the peak broadenings are unlikely due to the emergence of the new phase. Instead, in the sense that the linewidth of Raman peaks is a measure of electron-phonon coupling (EPC), the changes in FWHMs as a function of pressure imply an abrupt increase in EPC.^70,71 The anomalous FWHM jumps in P4 and P5 in the pressure range 7–11 GPa coincide with the c/a anomaly from XRD and the transition to a DSM state from DFT calculations. It is suggested that the increase in the EPC assists phonon-mediated renormalization of the electronic structure, and leads to the DSM state and elimination of surface states.⁷² The FWHM anomalies were absent in P6 and P7, which we have attributed to longitudinal vibrations, supporting the contention that the ETT signature is related more to the in-plane than the out-of-plane characteristics.⁶⁰ 

Figure 9 summarizes reported structural phase transitions, TI states, and reported anomalies of Bi₂Te₂, Bi₂Se₃, Sb₂Te₃, Sb₂Se₃, and Bi_1.5Sb_0.5Te_1.8Se_1.2 (from this study) as a function of pressure.^{14–20,22,27–29,50,55,73–77} The first structural phase transition of Bi_1.5Sb_0.5Te_1.8Se_1.2, from R$3¯$m to C2/m at 13–15 GPa, is consistent with that reported in other binary compounds such as Bi₂Te₃,^17,73 Bi₂Se₃,^16,50,74,77 and Sb₂Te₃.^15,19,75 Interestingly, the structural transition is observed at a higher pressure than the binary end members (∼7–10 GPa). The delayed structural transitions could be attributed to the atomic disorders within the pnictogen and chalcogen layers. Nam et al. have reported that with introduction of Sb, the pnictogen layer develops a random distribution of Bi/Sb atoms. In addition, disordered chalcogens could contribute to the disordered topography, possibly up to ∼1% of c-lattice constant.⁶³ The second structural phase transition of Bi_1.5Sb_0.5Te_1.8Se_1.2 to I4/mmm phase resembles the structural transition of Bi₂Se₃. Although the composition of the alloy is closest to Bi₂Te₃, introduction of Se resulted in the I4/mmm structure, instead of $Im3¯m$ structure. The C2/c phase was not as clearly observed in the Bi_1.5Sb_0.5Te_1.8Se_1.2 as in Bi₂Se₃.⁷⁷ 

Comparable with the delayed first structural transition, the c/a ratio anomaly and Raman FWHM anomaly (noted in Fig. 9 as “p” and “w,” respectively) of the Bi_1.5Sb_0.5Te_1.8Se_1.2 were observed at higher pressures than those of the binary compounds.^{16–19,26,27} The minimum in c/a ratio and FWHM anomaly have been observed below 5 GPa in case of binary compounds, compared to ∼7 GPa in Bi_1.5Sb_0.5Te_1.8Se_1.2. It is conceivable that the c/a anomalies occur when the vdW-like interlayer bonds are compressed enough to be as incompressible as intra-layer covalent bonds, and a further increase in pressure results in the structural phase transitions. In this sense, the c/a anomaly could be regarded as a precursor to the structural phase transition. The c/a anomaly is also closely accompanied by the Raman FWHM anomaly, which likely promotes the DSM state and dismantles the surface TI state. It could be considered that the TI state is encased in the R$3¯$m crystal phase, but only before the structural transition is imminent.

We have carried out a detailed study of the effects of hydrostatic pressure on the structural, vibrational, and topological properties of topological insulator Bi_1.5Sb_0.5Te_1.8Se_1.2 using diamond anvil cell experiments combined with ab initio density functional theory calculations. Two structural phase transitions were observed up to ∼30 GPa: from the ambient rhombohedral R$3¯$m phase to a monoclinic C2/m at ∼13 GPa, and to a disordered I4/mmm phase at ∼22 GPa. A series of electronic transitions were demonstrated within the R$3¯$m structure: (i) indirect-to-direct bulk band gap transition at ∼5.8 GPa, (ii) transition to 3D Dirac semimetal phase and vanishing of TI state at ∼8.2 GPa, and (iii) transition to topologically trivial metal at ∼12 GPa. The occurrence of the DSM state following c/a ratio anomaly and FWHM anomalies of in-plane Raman modes suggests contribution of electron–phonon coupling to the topological transition. By comparing with binary compounds, the aforementioned anomalies are also suggested to be precursors of the structural phase transition. Hydrostatic pressure has proven to be effective in modulating crystal symmetry and atomic bonds, and therefore capable of dramatically changing the electronic properties. This study on pressure-induced structural and topological transitions in the Bi_1.5Sb_0.5Te_1.8Se_1.2 provides deep insights toward understanding the surface state properties and their relationship to crystal structure.

See supplementary material for the details of Le Bail refinement of XRD patterns, calculated band dispersions of high pressure phases, and intensity analysis of Raman spectra.

R.J. would like to thank Dr. Ravindra Shinde for the help with surface state calculations. R.J. acknowledges the Department of Science and Technology (DST), New Delhi, India, for providing Fellowship under the INSPIRE programme with registration number IF150848. R.J. and A.K.S. would like to acknowledge support by the U.S. Army RDECOM-Pacific under Contract FA5209-16-P-0090. J.-S.K., J.-F.L., and D.A. would like to acknowledge support by the U.S. Army Research Office under Contract W911NF-13-1-0364. We also thank GSECARS of the Advanced Photon Source for the use of the X-ray diffraction facility for this project, Vitali Prakapenka for his assistance on the diffraction experiments, and Richard Roberts for scientific discussion.