Aiming at the problem of cracks in solar cells during the production process and under the condition of limited database of solar cell defects, the particle swarm optimization (PSO) is applied to optimize the support vector machines (SVM) to detect the surface cracks of solar cells. Firstly, in order to reduce the influence of uneven light distribution caused by electroluminescence (EL) detection in the image acquisition process, Retinex enhancement processing is performed on the image of the solar cell assembly. Secondly, in the frequency domain, the Gabor transform is used to extract the texture features of the image to obtain the crack feature. Finally, the texture features of each solar cell component are reduced by principal component analysis (PCA) and then they are input into the PSO_SVM system for classification and recognition. Using this method to experiment with 600 EL images of solar cells, only one image was detected by mistake, and the classification accuracy is 9933%. Comparing this algorithm with decision tree classification, extreme learning machine (ELM), convolutional neural network (CNN) and SVM algorithm, PSO_SVM achieves the highest recognition rate.