With the rapid proliferation of drones across various domains, aerial target detection has become increasingly crucial. However, the targets in aerial images present challenges such as scale variation, small size, and density, leading to suboptimal performance of current detectors on aerial images. Based on the aforementioned challenges, we design an efficient aerial target detection algorithm called DMFF-YOLO. Specifically, to address the issues of small target size and scale variation, we design the DMFF neck structure, adding a small target detection head to tackle the small target size problem, using the DMC module to fuse different scale features for enriching detailed information, and employing the DSSFF module to construct a scale sequence space to solve the target scale variation problem. In the network backbone, we employ RFCBAMConv modules as downsampling layers, which interact with receptive-field features to mitigate the information disparity caused by positional changes and outperform traditional convolutional layers. Finally, we design the Soft-NMS-CIoU module to address the issue of suppressing adjacent boxes due to dense targets. On the VisDrone dataset, compared to the original algorithm, our method reduces the number of parameters by 31.1% while achieving an 11.7% improvement in mAP50. Extensive experiments on the VisDrone, DOTA, and UAVDT datasets demonstrate that the proposed algorithm performs well in aerial image detection tasks.