Moving object detection is an essential component for security video surveillance system and other computer vision applications. Although the latest object detection methods get promising detection expectations, however, accurate detection is still a tricky problem due to various challenges such as aperture effects, illumination variations, camouflage issues and retention problems in unconstrained video environments. In this paper, we propose a brand-new theoretical framework for foreground object detection based on the stable spatial relationship between current pixel and the randomly selected pixels in current frame. Different from the existing methods which determine the moving object area by comparing each pixel value with its surrounding pixels or by comparing two pixel values occupying the same positions in adjacent frames, the proposed algorithm sets up a spatial sample set for each individual pixel and defines Spatial Sample Difference Consensus (SSDC), which denotes changes of stable spatial relationship rather than direct changes in pixel values. Thus, the proposed algorithm computes the SSDC between two adjacent frames to subtract the moving objects. The experiments on recent data-set in both indoor and outdoor surveillance video sequences show that the proposed method achieved promising performance after compared with several state-of-the art methods.