We address the problem of moving object detection in aerial video. Moving object detection in aerial video is still a challenging problem for the reason that when capturing the video the camera (or the platform) is moving all the time. As a result, the problem is detecting moving object from moving background which is much more difficult than the case that the background is constant. To this end, a novel approach is proposed in this paper. Moving object detection in stationary scene usually modeling the pixel value changes over time, but in aerial video the change does not have regular patterns. Therefore, we model the motion of the background rather than modeling the background directly. The optical flow between every two adjacent frames is computed first to get the motion information for each pixel. Based on this, we define a notion named ``pixel motion process" which means the motion changes (the optical flow value changes) of a particular pixel over time, and transfer the Gaussian mixture model framework used for modeling background in the stationary scene to model the background motion. The result is an accurate, adaptive and general background motion model which is used to detect foreground moving objects. Experimental results demonstrate the effectiveness of our approach.