Based on a generalized one-band Hubbard model, we study magnetic properties of Mott insulating states for ultracold spin-$\frac{3}{2}$ fermionic atoms in optical lattices. When the $s$-wave scattering lengths for the total spin $S=2,0$ satisfy conditions $a_2>a_0>0$, we apply a functional integral approach to the half filled case, where the spin-quadrupole fluctuations dominate. On a two-dimensional square lattice, the saddle-point solution yields a staggered spin-quadrupole ordering at zero temperature with symmetry breaking from SO(5) to SO(4). Both spin and spin-quadrupole static structure factors are calculated, displaying highly anisotropic spin antiferromagnetic fluctuations and antiferroquadrupole long-range correlations, respectively. When Gaussian fluctuations around the saddle point are taken into account, spin-quadrupole density waves with a linear dispersion are derived. Compared with the spin-density waves in the half filled spin-$\frac{1}{2}$ Hubbard model, the quadrupole density wave velocity is saturated in the strong-coupling limit, and there are no transverse spin-quadrupole mode couplings, as required by the SO(4) invariance of the effective action. Finally, in the strong-coupling limit of the model Hamiltonian, we derive the effective hyperfine spin-exchange interactions for the Mott insulating phases in the quarter filled and half filled cases, respectively.

• Received 21 August 2006

DOI:https://doi.org/10.1103/PhysRevB.74.174404