Cracks are primary threats to dam stability and require timely and precise identification and assessment. Dam cracks are typically morphologically complex and suffer from severe background interference. Current algorithms often struggle to strike a balance between detection accuracy and segmentation precision. To address this challenge, this paper proposes an improved YOLOv8m-based instance segmentation method for dam crack. The method integrates Cascaded Group Attention (CGA) to enhance the perception of global information and features of slender crack. It also employs Diverse Branch Block (DBB) technology to strengthen the convolutional network’s ability to extract subtle crack features without increasing inference costs. Additionally, the Minimum Point Distance IoU (MPDIoU) loss is introduced as an improved bounding box regression loss to enhance crack localization accuracy. Experiments are conducted on a dataset from the Xinfengjiang Dam in Heyuan City, Guangdong Province, China. The results show that the proposed method achieves 90.2% box mAP50, which is a 6.0% improvement over the original YOLOv8m. It also achieves 90.4% mask mAP50, which is a 7.5% improvement. Compared to other commonly used instance segmentation models, the proposed method demonstrates significant advantages in both detection and segmentation accuracy, offering a more advanced solution for dam safety monitoring.