Defect detection algorithms based on deep learning technology often need a large number of image samples to train the model because of many network parameters. However, in the process of industrial production, the number of defective products is very small, and collecting a large number of defect data images is time-consuming and laborious. To solve this problem, this paper proposes a lightweight defect detection algorithm based on multi model cascade. It adopts the training method of supervised learning, and can obtain better detection results through a small number of defect samples. Firstly, CBAM attention residual module is used to extract features instead of conventional convolution layer to focus on defect features and strengthen the characterization ability of network to defects; Secondly, the SE-FPN module is designed to promote the effective integration of features at all levels and improve the segmentation effect of network on defects, especially for small defects; Finally, in the training stage, the supervised learning method is used to train the multi model algorithm network proposed in this paper. The experimental results show that the detection accuracy of the proposed algorithm on KolektorSDD data set is as high as 99.28%, and the average detection time of each image is only 10.5ms. It not only fully meets the requirements of high precision and real-time in the industrial detection industry, but also realizes the accurate positioning of defect areas. Therefore, the research content of this paper is very suitable for application in the field of on-line detection of surface quality of industrial products.