Abstract This paper is about detecting and counting the passengers of a tracking vehicle using on-car monocular vision. By having a model of nearby vehicle occupants, intelligent reasoning systems of autonomous cars will be provided with this additional knowledge needed in emergency situations such as those that many philosophers have recently raised. The on-road Vehicle PassengEr Detection (ViPED) system is based on the human perception model in terms of spatio-temporal reasoning, namely the slight movements of passenger shape silhouettes inside the cabin. The main challenges we face are the low light conditions of the cabin (no feature points), the subtle non-rigid motions of the occupants (possible artifactual transitions), and the puzzling discrimination problem of back or front seat occupants (lack of depth information inside the cabin). To overcome these challenges, we first track the detected car windshield and find the optimal affine warp. The registered windshield images are preprocessed in order to extract a feature matrix, which serves as input to a Convolutional Neural Network (CNN) for inferring the number and position of passengers. We demonstrate that our low-cost sensor system is able to detect in most cases successfully all the passengers in preceding moving vehicles at various distances and occupancies. Metrics and datasets are included for possible community future work on this new challenging task.