Addressing the issues of significant scale differences in small targets and strong interference from complex backgrounds in the detection of surface defects on coal mine steel wire ropes, a deep learning detection algorithm based on an improved YOLOv11 is proposed. Firstly, a receptive field attention feature extraction module, C3k2_RFAConv, is designed to enhance feature extraction capabilities under complex textures by dynamically adjusting convolution kernel weights. Secondly, a deformable large kernel attention mechanism, D-LKA, is introduced at the feature fusion layer, combining the advantages of large receptive fields and deformable convolutions to precisely focus on defect areas. Additionally, DySample upsampling optimization is adopted to suppress background noise interference and reduce the loss of small target features. Finally, an Inner-WIoU loss function is proposed to optimize bounding box regression and improve the localization accuracy of irregular defects. Experimental results show that the improved algorithm achieves an accuracy rate of 83.2%, a recall rate of 78.1%, and an average precision of 82.1%, which are 3.1%, 4.6% and 2.6% higher than those of the benchmark model YOLOv11, respectively. It also outperforms comparative models such as Faster-RCNN and YOLOv8. In addition, visual analysis proves that the improved algorithm has a reduced missed detection rate, providing an effective technical solution for real-time monitoring of potential safety hazards in mining steel wire ropes.