To address issues such as misdetection and omission due to low light, image defocus, and worker occlusion in coal-rock image recognition, a new method called YOLOv8-Coal, based on YOLOv8, is introduced to enhance recognition accuracy and processing speed. The Deformable Convolution Network version 3 enhances object feature extraction by adjusting sampling positions with offsets and aligning them closely with the object’s shape. The Polarized Self-Attention module in the feature fusion network emphasizes crucial features and suppresses unnecessary information to minimize irrelevant factors. Additionally, the lightweight C2fGhost module combines the strengths of GhostNet and the C2f module, further decreasing model parameters and computational load. The empirical findings indicate that YOLOv8-Coal has achieved substantial enhancements in all metrics on the coal rock image dataset. More precisely, the values for AP50, AP50:95, and AR50:95 were improved to 77.7%, 62.8%, and 75.0% respectively. In addition, optimal localization recall precision (oLRP) were decreased to 45.6%. In addition, the model parameters were decreased to 2.59M and the FLOPs were reduced to 6.9G. Finally, the size of the model weight file is a mere 5.2 MB. The enhanced algorithm’s advantage is further demonstrated when compared to other commonly used algorithms.