The detection of grain pests is of great significance to grain storage. However, in practice, because the size of grain insects is too small to identify. In this paper, the feature fusion SSD(single shot multi-box detector) algorithm based on the Top-Down strategy was proposed. Firstly, the Top-Down module is used to fuse the output characteristics of conv4 and conv5, and the block 11 which is not conducive to small object detection is deleted. Secondly, K-means clustering algorithm is used to cluster prior bounding boxes and made them more suitable for grain pests, which improves the performance on small object detection of grain pests. Five methods were used to enhance the self-made dataset of grain pests, and the enhanced dataset reached 9990 images. Experiments on the enhanced dataset show that the optimized model achieves a mAP (mean Average Precision) 96.89% with detection speed of 0.040s per image. Compared with 95.45% of the mAP achieved by the original SSD algorithm, the proposed model has a great improvement on detection performance. Compared with two-stages Faster R-CNN (mAP is 90.53% and speed is 0.115s per image), YOLOv3, TDFSSD and EfficentDet (D2, D1), the speed and accuracy of the optimized SSD algorithm have obvious advantages. The experimental results show that the proposed SSD model has a good performance on small object detection of grain pests and has a certain guiding significance for subsequent grain pest image detection.