In order to account explicitly for the existence of long-periodic layered structures and the strong structural relaxations in the most common binary and ternary alloys of the Bi-Sb-Te-Se system, we have developed a one-dimensional cluster expansion (CE) based on first-principles electronic structure calculations, which accounts for the Bi and Sb bilayer formation. Excellent interlayer distances are obtained with a van der Waals density functional. It is shown that a CE solely based on pair interactions is sufficient to provide an accurate description of the ground-state energies of Bi-Sb-Te-Se binary and ternary systems without making the data set of ab initio calculated structures unreasonably large. For the binary alloys $A_{1-x}{Q}_x$ ($A=\text{Sb}$, Bi; $Q=\text{Te}$, Se), a ternary CE yields an almost continuous series of (meta)stable structures consisting of consecutive $A$ bilayers next to consecutive $A_2{Q}_3$ for $0<x<0.6$. For $x>0.6$, the binary alloy segregates into pure $Q$ and $A_2{Q}_3$. The Bi-Sb system is described by a quaternary CE and is found to be an ideal solid solution stabilized by entropic effects at $T\ne 0$ K but with an ordered structure of alternating Bi and Sb layers for $x=0.5$ at $T=0$ K. A quintuple CE is used for the ternary Bi-Sb-Te system, where stable ternary layered compounds with an arbitrary stacking of ${\text{Sb}}_2$${\text{Te}}_3$, ${\text{Bi}}_2$${\text{Te}}_3$, and Te-Bi-Te-Sb-Te quintuple units are found, optionally separated by mixed Bi/Sb bilayers. Electronic properties of the stable compounds were studied taking spin-orbit coupling into account.

• Received 12 September 2013
• Revised 13 February 2014

DOI:https://doi.org/10.1103/PhysRevB.89.054106