Drilling operations are an essential part of furniture from MDF laminated boards required for product assembly. Faults in the process might introduce adverse effects to the furniture. Inspection of the drilling quality can be challenging due to a big variety of board surface textures, dust, or woodchips in the manufacturing process, milling cutouts, and other kinds of defects. Intelligent computer vision methods can be engaged for global contextual analysis with local information attention for automated object detection and segmentation. In this paper, we propose blind and through drilled holes segmentation on textured wooden furniture panel images using the UNet encoder-decoder modifications enhanced with residual connections, atrous spatial pyramid pooling, squeeze and excitation module, and CoordConv layers for better segmentation performance. We show that even a lightweight architecture is capable to perform on a range of complex textures and is able to distinguish the holes drilling operations’ semantical information from the rest of the furniture board and conveyor context. The proposed model configurations yield better results in more complex cases with a not significant or small bump in processing time. Experimental results demonstrate that our best-proposed solution achieves a Dice score of up to 97.89% compared to the baseline U-Net model’s Dice score of 94.50%. Statistical, visual, and computational properties of each convolutional neural network architecture are addressed.