Aiming at the problems of leakage and wrong detection when the YOLOv8 algorithm is applied to the surface defect detection of strip steel, an improved YOLOv8 algorithm is proposed. For the labels of small targets in the dataset, Normalized Gaussian Wasserstein Distance (NWD) is added on top of the original lossy CIOU, which improves the model's ability to detect defects of small targets; Focal Modulation is used to replace the spatial pooling pyramid of the YOLOv8 model, which improves the expression ability of multi-scale features while lightweighting; Mobile Inverted Bottleneck Conv (MBConv) is used to replace the Conv in C2f to construct a new module C2f-MB, and at the same time replace the original C2f-MB with C2f-MB. MB to replace the original C2f module with C2f-MB, which enhances the feature expression ability and multi-scale feature fusion ability; the Convolutional Block Attention Module (CBAM) is added in the backbone part to suppress the background interference, which can better capture the global information and improve the feature extraction ability of the backbone part. Experiments show that the improved YOLOv8 algorithm improves mAP@50 by 3 percentage points while decreasing the computation amount, which significantly improves the problems of missed detection and wrong detection.