Efficient operation of new energy storage systems relies heavily on accurately estimating the state of charge (SOC) of lithium-ion batteries. This paper introduces a novel approach that incorporates fractional calculus to improve SOC estimation for lithium batteries. Firstly, a second-order RC model based on fractional-order calculus theory was developed to model the lithium battery characteristics. Then perform a pulse characterization test to obtain the battery terminal voltage, and complete parameter identification based on the recursive least squares method with adaptive forgetting factor (AFFRLS). In addition, the proposed algorithm based on fractional unscented Kalman filter (FUKF) is applied to estimate SOC in battery discharge experiments. Finally, compared the three prediction indicators of maximum absolute error (MAE), average absolute error (AAE) and root mean square error (RMSE) with the unscented Kalman filter (UKF), the method introduced in this paper exhibits greater precision. The results show that the MAE of the FUKF algorithm in estimating SOC is less than 0.02, and it has higher accuracy than UKF algorithm.