Target detection based on deep learning is developing rapidly. However, small target detection is still a challenge. In this paper, a simple and efficient network for small target detection is proposed. We put forward to improve the detection performance of the small targets in three aspects. First, as the contextual information is important to detect the small targets, we proposed to use “dilated module” to expand the receptive field without loss of resolution or coverage. Second, we applied feature fusion in different dilated modules to improve the ability of the network in detecting small targets. Finally, we used “passthrough module” to get the finer-grained information from the earlier layer and combined it with the semantic information from the deeper layer. To improve the detection speed of the network, it is proposed to use $1\times 1$ convolution to reduce the dimension of the network. We composed small vehicle dataset based on VEDAI dataset and DOTA dataset, respectively, and also analyzed the distribution of the small targets in each dataset. To evaluate the performance of the proposed network, we trained the model on the dataset above and compared with the state-of-the-art target detection algorithms, our approach achieved 80.16% average precision (AP) on VEDAI dataset and 88.63% AP on DOTA dataset and the frames per second (FPS) is 75.4. The AP of our network is much better than the result of the tiny YOLO V3 and is nearly the same as the result of the YOLO V3. However, the FPS of our network is almost the same as that of the tiny YOLO V3.