A theory of selective reflection (SR) of a light beam from the interface of a dielectric medium and a dense atomic vapor is presented. Following another earlier work on this subject [J. Guo et al., Opt. Commun. 110, 732 (1994)], we calculate the atomic density matrix and the transmitted field in the vapor self-consistently. We extend our previous theory to include an attractive atom-dielectric wall interaction and multiple atomic transitions. We examine the limits at which the conventional SR theories, which are based on the assumption of a plane-wave field in the vapor, start to fail. It is shown that at or above vapor densities of order ${10}^{17}$/${\mathrm{cm}}^3$, the frequency shifts of the SR line shape due to the local-field correction (Lorentz-Lorenz shift), the atom-wall interaction, and the nonexponential attenuation of the field in the vapor can have comparable orders of magnitude, and the proper determination of all these effects is essential for calculating the correct SR signals. A comparison of the theoretical SR spectra with an experimental result for a Rb vapor with a density of order ${10}^{17}$/${\mathrm{cm}}^3$ is also presented. © 1996 The American Physical Society.

• Received 17 August 1995

DOI:https://doi.org/10.1103/PhysRevA.53.1130