This paper proposes a unified framework for spatiotemporal segmentation of video sequences. A Bayesian network is presented to model the interactions among the motion vector field, the intensity segmentation field, and the video segmentation field. The notions of distance transformation and Markov random field are used to express spatiotemporal constraints. Given consecutive frames, an optimization method is proposed to maximize the conditional probability density of the three fields in an iterative way. Experimental results show that the approach is robust and generates spatiotemporally coherent segmentation results.