We study the quantum phases driven by interaction in a semimetal with a quadratic band touching at the Fermi level. By combining the density matrix renormalization group (DMRG), analytical power expanded Gibbs potential method, and the weak coupling renormalization group, we study a spinless fermion system on a checkerboard lattice at half-filling, which has a quadratic band touching in the absence of interaction. In the presence of strong nearest-neighbor ($V_1$) and next-nearest-neighbor ($V_2$) interactions, we identify a site nematic insulator phase, a stripe insulator phase, and a phase separation region, in agreement with the phase diagram obtained analytically in the strong coupling limit (i.e., in the absence of fermion hopping). In the intermediate interaction regime we establish a quantum anomalous Hall phase in the DMRG as evidenced by the spontaneous time-reversal symmetry breaking and the appearance of a quantized Chern number $C=1$. For weak interaction we utilize the power expanded Gibbs potential method that treats $V_1$ and $V_2$ on equal footing, as well as the weak coupling renormalization group. Our analytical results reveal that not only the repulsive $V_1$ interaction, but also the $V_2$ interaction (both repulsive and attractive), can drive the quantum anomalous Hall phase. We also determine the phase boundary in the $V_1\text{−}{V}_2$ plane that separates the semimetal from the quantum anomalous Hall state. Finally, we show that the nematic semimetal, which was proposed for $|V_2|\gg V_1$ at weak coupling in a previous study, is absent, and the quantum anomalous Hall state is the only weak coupling instability of the spinless quadratic band touching semimetal.

• Received 19 May 2018

DOI:https://doi.org/10.1103/PhysRevB.98.125144