We study collective behavior of magnetic adatoms randomly distributed on the surface of a topological insulator. Interactions of an ensemble of adatoms are frustrated, as the RKKY-type interactions of two adatom spins depend on the directions of spins relative to the vector connecting them. We show that at low temperatures the frustrated RKKY interactions give rise to two phases: an ordered ferromagnetic phase with spins pointing perpendicular to the surface, and a disordered spin-glass-like phase. The two phases are separated by a quantum phase transition driven by the magnetic exchange anisotropy. The ordered phase breaks time-reversal symmetry spontaneously, driving the surface states into a gapped state, which exhibits an anomalous quantum Hall effect and provides a realization of the parity anomaly. We find that the magnetic ordering is suppressed by potential scattering.

• Received 4 October 2010

DOI:https://doi.org/10.1103/PhysRevLett.106.136802