In the fault types of permanent magnet synchronous motors (PMSM), inter-turn short circuit (ITSC) faults are relatively common, making the accurate extraction of fault features particularly significant. However, during fault feature extraction, modal mixing often occurs. In order to accurately extract the fault features of vibration signals in permanent magnet synchronous motor (PMSM) when inter-turn short circuit (ITSC) occurs, proposes an adaptive nonlinear signal processing method based on particle swarm optimized variational mode decomposition (PSO-VMD). Firstly, particle swarm optimization (PSO) is used to find the optimal number of decomposition layers and quadratic penalty factor for variational modal decomposition (VMD) to obtain the optimal decomposition model. Secondly, the optimal decomposition model is used to decompose the motor vibration signals to obtain a series of intrinsic mode functions (IMF). After that, the variance contribution rate (VCR) of each IMF is calculated, and the cumulative variance contribution rate (C-VCR) is further calculated to filter out the IMF that contain fault signature information. Finally, the filtered IMF are analyzed by applying the Hilbert transform (HT), and the three-dimensional time-frequency diagrams are used to output the time, the instantaneous frequency and the amplitude to complete the fault feature extraction. In order to verify the validity and accuracy of the proposed method, an experimental platform of the ITSC in PMSM was built, and the proposed method was used to process the measured signals. The experimental results show that the proposed PSO-VMD method effectively improves the phenomenon of modal mixing, can more accurately extract fault features, and has better engineering applicability.