Aiming at the problems of irregular morphology and suboptimal detection performance of surface defect on aluminum ingot alloys, an improved YOLOv5-based defect detection method is proposed. Firstly, The Res2Net feature extraction network block is employed to replace the CSPDarknet53 module of the baseline model, which can effectively detect the multi-scale defect. Secondly, the CBAM convolutional attention module is introduced into the backbone network of YOLOv5 to enhance the representational ability of defect features. Finally, the over-parameterized reparameterization convolutional blocks are used to substitute for the 3×3 convolutional blocks in the backbone and neck networks so as to reduce the model′s inference latency. Experimental results compared with the traditional target detection methods demonstrate the improved method achieves a mAP of 75.8% for defect detection, which is a significant improvement both in detection accuracy and inference speed, and can well satisfy the tasks and demands of practical industrial production.