We study the magnetic phase diagram of spin-$\frac{3}{2}$ fermions in a spatially anisotropic square optical lattice at quarter filling (corresponding to one particle per lattice site). In the limit of the large on-site repulsion the system can be mapped to the so-called Sp$(N)$ Heisenberg spin model with $N=4$. We analyze the Sp$(N)$ spin model with the help of the large-$N$ field-theoretical approach and show that the effective theory corresponds to the Sp$(N)$ extension of the ${\mathit{CP}}^{N-1}$ model, with the Lorentz invariance generically broken. We obtain the renormalization flow of the model couplings and show that although the Sp$(N)$ terms are seemingly irrelevant, their presence leads to a renormalization of the ${\mathit{CP}}^{N-1}$ part of the action, driving a phase transition. We further consider the influence of the external magnetic field (the quadratic Zeeman effect) and present the qualitative analysis of the ground-state phase diagram.

• Received 3 August 2010

DOI:https://doi.org/10.1103/PhysRevB.83.014418