The widespread application of drone technology has raised security concerns, as unauthorized drones can lead to illegal intrusions and privacy breaches. Traditional detection methods often fall short in balancing performance and lightweight design, making them unsuitable for resource-constrained scenarios. To address this, we propose the IASL-YOLO algorithm, which optimizes the YOLOv8s model to enhance detection accuracy and lightweight efficiency. First, we design the CFE-AFPN network to streamline the architecture while boosting feature fusion capabilities across non-adjacent layers. Second, we introduce the SIoU loss function to address the orientation mismatch issue between predicted and ground truth bounding boxes. Finally, we employ the LAMP pruning algorithm to compress the model. Experimental results on the Anti-UAV dataset show that the improved model achieves a 2.9% increase in Precision, a 6.8% increase in Recall, and 3.9% and 3.8% improvements in mAP50 and mAP50-95, respectively. Additionally, the model size is reduced by 75%, the parameter count by 78%, and computational workload by 30%. Compared to mainstream algorithms, IASL-YOLO demonstrates significant advantages in both performance and lightweight design, offering an efficient solution for drone detection tasks.