This paper proposes a joint random field (JRF) model for moving vehicle detection in video sequences. The JRF model extends the conditional random field (CRF) by introducing auxiliary latent variables to characterize the structure and evolution of visual scene. Hence detection labels (e.g. vehicle/roadway) and hidden variables (e.g. pixel intensity under shadow) are jointly estimated to enhance vehicle segmentation in video sequences. Datadependent contextual constraints among both detection labels and latent variables are integrated during the detection process. The proposed method handles both moving cast shadows/lights and background illumination variations. Computationally efficient algorithm has been developed for realtime vehicle detection in video streams. Experimental results show that the approach effectively deals with various illumination conditions and robustly detects moving vehicles even in grayscale video.