The characteristics of phase transformation in nanocrystalline alloys were studied both theoretically and experimentally from the viewpoint of thermodynamics. With a developed thermodynamic model, the dependence of phase stability and phase transformation tendency on the temperature and the nanograin size were calculated for the nanocrystalline Sm₂Co₁₇ alloy. It is thermodynamically predicted that the critical grain size for the phase transformation between hexagonal and rhombohedral nanocrystalline Sm₂Co₁₇ phases increases with increasing temperature. When the grain size is reduced to below 30 nm, the hexagonal Sm₂Co₁₇ phase can stay stable at room temperature, which is a stable phase only at temperatures above 1520 K in the conventional polycrystalline alloys. A series of experiments were performed to investigate the correlation between the phase constitution and the grain structure in the nanocrystalline Sm₂Co₁₇ alloy with different grain size levels. The experimental results agree well with the thermodynamic predictions of the grain-size dependence of the room-temperature phase stability. It is proposed that at a given temperature the thermodynamic properties, as well as the phase stability and phase transformation behavior of the nanocrystalline alloys, are modulated by the variation of nanograin size, i.e. the grain size effects on the structure and energy state of the nanograin boundaries.