We use exact diagonalization to study the quantum phases and phase transitions when a single species of fermionic atoms at a Landau level filling factor ${\nu }_f=1$ in a rotating trap interact through a $p$-wave Feshbach resonance. We show that under a weak pairing interaction, the system undergoes a second-order quantum phase transition from a ${\nu }_f=1$ fermionic integer quantum Hall (FIQH) state at positive detuning, to a ${\nu }_b=\frac{1}{4}$ bosonic fractional quantum Hall (BFQH) state at negative detuning. However, when the pairing interaction increases, a new phase between them emerges, corresponding to a fraction of fermionic atoms staying in a coherent superposition of a bosonic molecule state and an unbound pair. The phase transition from the FIQH phase to the new phase is of second order and that from the new phase to BFQH phase is of first order.

• Received 18 April 2017

DOI:https://doi.org/10.1103/PhysRevB.95.241106