The vibration signal of planetary gearboxes is complex with timevarying transmission paths, which makes fault feature extraction of planetary gearboxes difficult. For this issue, a fault feature extraction method based on windowed vibration separation and variational mode decomposition (VMD) is proposed. Firstly, the angle synchronous average is used to eliminate asynchronous components. Then, the windowed vibration separation is implemented. The parameter K is determined by observing the center frequency, and the VMD is applied to separated vibration signals to select the component containing fault feature. Finally, the selected feature component is demodulated by Hilbert demodulation analysis to achieve the fault feature extraction. By analyzing the measured signal of planet gear root crack fault, it is proved that this method can effectively extract the fault feature of the planet gear.