This paper takes the planetary gear transmission system with high incidence of faults as the object, a fault diagnosis method based on variational mode decomposition (VMD) and particle swarm optimization (PSO) to optimize support vector machine (SVM) is presented. Firstly, the signal is decomposed by VMD, the decomposed components are processed by improved wavelet method, and the processed components are reconstructed to highlight the signal. The weak information of SVM is extracted. Then, the sample entropy and root mean square error of the processed vibration signal are extracted, and the input matrix is formed. Finally, PSO is introduced to optimize the key parameters of SVM, and the extracted eigenvectors are input into PSO-SVM for training and recognition. The method is applied to the planetary gear crack fault, the solar gear tooth fault and the planetary gear bearing fault signal obtained by the planetary transmission test platform. The effectiveness of the method is verified by multi-dimensional comparison.