During the tobacco leaf sorting process, manual factors can lead to non-compliant tobacco leaf loosening, resulting in low-quality tobacco leaf sorting such as mixed leaf parts, mixed grades, and contamination with non-tobacco related materials. Given the absence of established methodologies for monitoring and evaluating tobacco leaf sorting quality, this paper proposes a YOLO-TobaccoStem-based detection model for quantifying tobacco leaf loosening rates. Initially, a darkroom image acquisition system was constructed to create a stable monitoring environment. Subsequently, modifications were made to YOLOv8 to improve its multi-scale object detection capabilities. This was achieved by adding layers for detecting smaller objects and integrating a weighted bi-directional feature pyramid structure to reconstruct the feature fusion network. Additionally, a loss function with a monotonic focusing mechanism was introduced to increase the model’s learning capacity for difficult samples, resulting in a YOLO-TobaccoStem model more suitable for detecting tobacco stem objects. Lastly, a tobacco leaf loosening rate detection algorithm was formulated. The results from the YOLO-TobaccoStem were input into this algorithm to determine the compliance of the tobacco leaf loosening rate. The detection method achieved an F1-Score of 0.836 on the test set. Experimental results indicate that the proposed tobacco leaf loosening rate detection method has significant practical application value, enabling effective monitoring and evaluation of tobacco leaf sorting quality, thereby further enhancing the quality of tobacco leaf sorting.