Automating the process of detecting pavement cracks became a challenge mission. In the last few decades, many methods were proposed to solve this problem. The reason is that maintaining a stable condition of roads is essential for the safety of people and public properties. It was reported that maintaining one mile of roads in New York City in the USA might cost from four to ten thousand dollars. In this paper, we explore our initial idea of developing a lightweight Convolutional Neural Network (CNN or ConvNet) model that can be used to detect pavement cracks. The proposed CNN was trained using the AigleRN data set, which contains 400 images of road cracks of 480×320 resolution. The proposed lightweight CNN architecture performed a better fitting to the image data set due to the reduction in the number of parameters. The proposed CNN was capable of detecting cracks with a various number of sample images. We simulated the CNN architecture over different sizes of training/testing (i.e., 90/10, 80/20, and 70/30) data sets for 11 runs. The obtained results show that 90/10 data division for training and testing is outperformed other categories with an average accuracy of 97.27%.