We study the combined effects of fluctuations and frustration in frustrated antiferromagnets described by the spin-1/2 easy-axis XXZ model with transverse magnetic fields, and propose its promising quantum simulation with a coherently coupled binary mixture of Fermi atoms in a triangular optical lattice. We perform a cluster mean-field plus scaling analysis to show that the ground state exhibits several nontrivial magnetic phases and a spin reorientation transition caused by the quantum order-by-disorder mechanism. Moreover, we find from Monte Carlo simulations that thermal fluctuations induce an unexpected coexistence of Berezinskii-Kosterlitz-Thouless physics and long-range order in different correlators. These predictions, besides being relevant to present and future experiments on triangular antiferromagnetic materials, can be tested in the laboratory with the combination of the currently available techniques for cold atoms.

• Received 27 February 2019
• Revised 9 September 2019

DOI:https://doi.org/10.1103/PhysRevB.100.140410