Rotating machinery is the most widely used mechanical equipment in production, and its transmission system is an important part of the rotating machinery. Whether the rotating machinery transmission system running in safe and stable state has an important impact on the economy and society, so, researched on running stability deterioration of rotating machinery is significant. Rotor test rig has typical transmission structure of rotating machinery, and feature parameter extraction method faced to running stability deteriorationof its transmission system has been proposed.This paper describes the evolutionary trajectory of stable running state deterioratedto the unstable running state, and deterioration evolutionary matrix has been proposed based on higherorder cumulant method. A feature parameter extractionhas been used to depict evolutionary track faced to running stability deterioration of rotating machinery based on characterization parameters of the state deterioration trend such as mean value of stability deterioration and varianceof stability deterioration. The vibration data of the actual operating state of rotor test rigare collected, and experimental verification is carried out based on the method.The results show that this method can describe the overall trend of the operation statesface to the transmission system of the rotating machinery.