Taiwan’s economy mainly relies on the export of agricultural products. If even the suspicion of a pest is found in the crop products after they are exported, not only are the agricultural products returned but the whole batch of crops is destroyed, resulting in extreme crop losses. The species of mealybugs, Coccidae, and Diaspididae, which are the primary pests of the scale insect in Taiwan, can not only lead to serious damage to the plants but also severely affect agricultural production. Hence, to recognize the scale pests is an important task in Taiwan’s agricultural field. In this study, we propose an AI-based pest detection system for solving the specific issue of detection of scale pests based on pictures. Deep-learning-based object detection models, such as faster region-based convolutional networks (Faster R-CNNs), single-shot multibox detectors (SSDs), and You Only Look Once v4 (YOLO v4), are employed to detect and localize scale pests in the picture. The experimental results show that YOLO v4 achieved the highest classification accuracy among the algorithms, with 100% in mealybugs, 89% in Coccidae, and 97% in Diaspididae. Meanwhile, the computational performance of YOLO v4 has indicated that it is suitable for real-time application. Moreover, the inference results of the YOLO v4 model further help the end user. A mobile application using the trained scale pest recognition model has been developed to facilitate pest identification in farms, which is helpful in applying appropriate pesticides to reduce crop losses.