Hexagonal lattice systems (e.g., triangular, honeycomb, kagome) possess a multidimensional irreducible representation corresponding to $d_{x^2-y^2}$ and $d_{xy}$ symmetry. Consequently, various unconventional phases that combine these $d$-wave representations can occur, and in so doing may break time-reversal and spin-rotation symmetries. We show that hexagonal lattice systems with extended repulsive interactions can exhibit instabilities in the particle-hole channel to phases with either $d_{x^2-y^2}+d_{xy}$ or $d+id$ symmetry. When lattice translational symmetry is preserved, the phase corresponds to nematic order in the spin channel with broken time-reversal symmetry, known as the $\beta$ phase. On the other hand, lattice translation symmetry can be broken, resulting in various $d_{x^2-y^2}+d_{xy}$ density wave orders. In the weak-coupling limit, when the Fermi surface lies close to a van Hove singularity, instabilities of both types are obtained in a controlled fashion.

• Received 19 April 2013

DOI:https://doi.org/10.1103/PhysRevB.88.205121