Moving object detection is one of the key technologies for intelligent video monitoring systems. For real-time detection of moving object in the surveillance scene, the general and simple method is based on background image difference. However, it requires the accurate current background image and the approach for automatic background updating along with the illumination variance is difficult to design and implement. This limits its applications. To solve the above problem, a new self-adaptive background approximating and updating algorithm based on optical flow theory is presented for the traffic surveillance scene in this paper. To detect the moving regions of interest in the scene, the difference image between the current frame and the updating background is first obtained by using a color image difference model, and then a self-adaptive thresholding segmentation method for moving object detection based on the Gaussian model is developed and implemented. Moreover, an effective shadow-eliminating algorithm based on contour information and color features is developed. Experimental results demonstrate that the proposed background updating method can update the background exactly and rapidly along with the variance of illumination, the self-adaptive thresholding segmentation method based on the Gaussian model can extract the moving object regions accurately and completely, and the shadow can be eliminated accurately. This is the foundation for further objects recognition and understanding.