In video surveillance, detection of moving objects from an image sequence is very important for object tracking, activity recognition and behavior understanding. The conventional background subtraction suffers from slow updating of environmental changes, and temporal difference cannot accurately extract the moving object boundaries. In this paper, a Fourier reconstruction scheme for motion detection is proposed. A series of consecutive 2D spatial images along the time axis are first reorganized as a series of 2D spatial–temporal images along a spatial axis. In each of the 2D spatial–temporal images, a static background region forms a vertical line pattern, and a moving object creates an irregular, non-vertical structure in the image. Fourier transforms are applied to remove the vertical line pattern (i.e. the background) and retain only the foreground in the reconstructed image. The proposed method is a global approach that identifies the moving objects based on structural variations in the whole patterned image. It is therefore very robust to accommodate noise and local gray-level variations. It can well extract the shapes of foreground objects at various moving speeds, and is very responsive to dynamic environments. High computational cost is the major drawback of the proposed method. However, it can still achieve 11 frames per second for small images of size 150 × 200.