In view of the complex background of solar cell image, changeable defect morphology and large scale difference, a method of solar cell defect detection based on SimAM-Ada Pool YOLOv5 algorithm was proposed. First, deformable convolution is incorporated into the CBL module to achieve adaptive learning of feature scales and perceptual field sizes; then, Ada Pool is incorporated into the SPP module to increase the degree of defect information retention; finally, the feature extraction capability of the model is further improved by introducing the SimAM attention mechanism. To further optimize and improve the YOLOv5 algorithm, the Mosaic and MixUp fusion data enhancement, K-means++ clustering anchor box algorithm, CIOU loss function, and Hard-Swish activation function are used to enhance the performance of the improved model. The experimental results show that the improved YOLOv5 algorithm achieves 89.86% detection mAP on the solar cell electroluminescence image dataset, which is 8.07% higher than the mAP of the original algorithm, with a speed of 37.92 fps, and can complete the solar cell defect detection task more accurately while meeting the real-time requirements.