In the past few decades, tremendous efforts have been made toward understanding the exotic physics emerging from competition between various ordering tendencies in strongly correlated systems. Employing state-of-the-art quantum Monte Carlo simulation, we investigate an interacting $\mathrm{SU}(N)$ fermionic model with varying interaction strength and value of $N$, and we unveil the ground-state phase diagram of the model exhibiting a plethora of exotic phases. For small values of $N$—namely, $N=2$, 3—the ground state is an antiferromagnetic (AFM) phase, whereas in the large-$N$ limit, a staggered valence bond solid (VBS) order is dominant. For intermediate values of $N$ such as $N=4$, 5, remarkably, our study reveals that distinct VBS orders appear in the weak and strong coupling regimes. More fantastically, the competition between staggered and columnar VBS ordering tendencies gives rise to a Mott insulating phase without spontaneous symmetry breaking (SSB), existing in a large interacting parameter regime, which is consistent with a gapped quantum spin liquid. Our study not only provides a platform to investigate the fundamental physics of quantum many-body systems—it also offers a novel route toward searching for exotic states of matter such as quantum spin liquid in realistic quantum materials.

• Received 29 November 2022
• Revised 26 November 2023
• Accepted 1 December 2023

DOI:https://doi.org/10.1103/PhysRevLett.132.036704