Target detection in aerial remote sensing images is crucial for environmental monitoring and military reconnaissance. Infrared imaging is the main means of target detection in low-light or adverse weather conditions, but its detection performance is limited by the scarce annotated samples, which are costly to produce. In contrast, visible light images are usually abundant and well-annotated. This study aims to enhance the detection performance on unlabeled infrared images using unsupervised domain adaptation technique with the help of annotated visible light images. Since infrared and visible light images have different imaging principles, conventional domain adaptation techniques often lead to negative transfer. In this article, we develop a new domain adaptation framework for infrared target detection at both the region level and sample level to tackle the problem of negative transfer. To address region negative transfer from visible light images to infrared images, we design a RPMS-DA module to make the transfer focus on important regions and suppress noises. Besides, a SWMS-DA module is designed to accommodate those samples too difficult or too easy to transfer adaptively. Finally, the proposed region and sample level domain adaptation framework is realized based on the advanced YOLOv7 one-stage detection backbone. We conducted comprehensive experiments based on the VEDAI and DroneVehicle aerial remote sensing datasets, and the experimental results demonstrate that our algorithm achieves better performance than those state-of-the-art unsupervised domain adaptation target detection algorithms. Our algorithm achieves a good balance between accuracy and complexity.