Aiming at the problems of complex image background and poor recognition of small defect targets in the current insulator surface defect recognition, a transmission line insulator surface defect recognition algorithm based on YOLOv8 is proposed. Firstly, the CAF module is introduced in the backbone network to enhance the model's analysis of complex image scenes and enhance the ability to extract global and local features; secondly, the GD mechanism is added to the neck network of the model to reduce the loss of information in the feature fusion process and improve the small target detection ability; finally, the ATFL classification loss function is used to weaken the interference of complex background on small target detection, and the PIOU bounding box loss function is introduced to improve the recognition accuracy and accelerate the model convergence speed. Experimental results show that the mAP50 of the algorithm reaches 94.1%, the precision rate reaches 92.5%, and the recall rate reaches 91.3%, which are 3.1%, 0.7%, and 3.9% higher than the baseline model, respectively, and the comprehensive performance is better than the recent YOLOv9s, YOLOv10s and other representative algorithms.