We derive the effective action for the collective spin modes in iron-based superconductors. We show that, due to the orbital-selective nature of spin fluctuations, the magnetic and nematic instabilities are controlled by the degrees of orbital nesting between electron and hole pockets. Within a prototypical three-pocket model the hole-electron orbital mismatch is found to boost spin-nematic order. This explains the enhancement of nematic order in FeSe as compared to 122 compounds, and its suppression under pressure, where the emergence of the second hole pocket compensates the orbital mismatch of the three-pocket configuration.

• Received 30 July 2017
• Revised 18 February 2018

DOI:https://doi.org/10.1103/PhysRevB.97.121109