The effects that nanometer-sized matrices have on the properties of molecules encapsulated within the nanomatrix are not fully understood. In this work, dye-doped silica nanoparticles were employed as a model for studying the effects of a nanomatrix on the fluorescence quantum yield of encapsulated dye molecules. Two types of dye molecules were selected based on their different responses to the surrounding media. Several factors that affect fluorescence quantum yields were investigated, including aggregation of dye molecules, diffusion of atmospheric oxygen, concentration of dye molecules, and size of the nanomatrix. The results showed that the silica nanomatrix has a varied effect on the fluorescence quantum yield of encapsulated dye molecules, including enhancement, quenching and insignificant changes. Both the properties of dye molecules and the conditions of the nanomatrix played important roles in these effects. Finally, a physical model was proposed to explain the varied nanomatrix effects on the fluorescence quantum yield of encapsulated dye molecules.