To address the issue of output saturation caused by higher-order term constraints, we propose a novel unsaturated tri-stable second-order stochastic resonance (UTSOSR) system that leverages the excellent anti-saturation properties of piecewise potential function. First, simulation experiments verified that this system can significantly mitigate the output saturation problem of classical tri-stable second-order stochastic resonance system. Next, based on the adiabatic approximation theory, we derived the steady-state probability density, mean first-passage time, and spectral amplification factor (SA) of the UTSOSR system. By analyzing the influence of various system parameters on these performance metrics, we can further explore the system’s dynamic behavior in greater depth. Subsequently, using the SA and the signal-to-noise ratio gain (Gsnr) as evaluation metrics, numerical simulations were conducted to verify the superior signal enhancement and noise robustness performance of the UTSOSR system. Additionally, to achieve superior output performance, we combined maximum correlated kurtosis deconvolution (MCKD) with the UTSOSR system, proposing the MCKD-UTSOSR method for extracting target signal features. Finally, a combined approach using genetic algorithm and variable step-size grid optimization algorithm is employed to identify the optimal parameters for the MCKD-UTSOSR method in bearing fault diagnosis. The data analysis results indicate that compared to other methods, the MCKD-UTSOSR method improved the signal-to-noise ratio by 1.128 9~23.585 4 dB and the spectral peak value by 88.423~7 488.118 133. This provides an innovative and reliable solution for efficient signal processing and fault detection in practical engineering applications.