Nanoscale hydroxyapatite (nano-HAP) has been reported to exhibit anti-cancer effect on several human cancers, but the molecular mechanism of which remains unclear. The aim of this study was to explore the mechanisms by investigating the effects of nano-HAP on human gastric cancer SGC-7901 cells. Our results showed that nano-HAP significantly reduced cell viability, and induced apoptosis in SGC-7901 cells characterized by hypodiploid DNA contents, morphological changes and DNA fragmentation. The increase in apoptosis was accompanied with the increased expression of Bax, a pro-apoptotic protein, and decreased expression of Bcl-2, an anti-apoptotic protein, the decrease of mitochondrial membrane potential and the release of cytochrome c from mitochondria into cytosol. Furthermore, the activation of caspases-3, and -9, but not activation of caspases-8 was induced by nano-HAP. Z-VAD-fmk, a universal caspase inhibitor, dose-dependently inhibited nano-HAP-induced apoptosis. This study demonstrates that nano-HAP inhibits the proliferation of SGC-7901 cells by inducing apoptosis, and the apoptotic pathway of nano-HAP-induced apoptosis is mediated through the mitochondrial-dependent and caspase-dependent pathway.