To address the limitation of existing direction-of-arrival (DOA) estimation algorithms in simultaneously suppressing alpha-stable distribution noise and Gaussian colored noise, a DOA estimation method based on fractional-order cumulants and a weighted hyperbolic composite-function smoothed l0-norm is proposed. First, by exploiting the semi-invariant property of fractional-order cumulants and their insensitivity to both alpha-stable and Gaussian distributions, the adverse effects of alpha-stable noise and Gaussian colored noise are effectively suppressed. Then, the fractional-order cumulant matrix is vectorized, and a sparse DOA reconstruction model based on fractional-order cumulants is formulated. Subsequently, a hyperbolic composite function is constructed to provide a smooth approximation of the l0-norm, where the approximation parameter is adaptively adjusted to balance smoothness and steepness. Meanwhile, a weighting matrix is introduced to enhance the coefficients corresponding to the true source locations while suppressing the remaining entries, thereby further emphasizing the sparse structure of the solution. Based on this formulation, a two-layer iterative optimization strategy combining gradient descent and projection is employed to solve the weighted smoothed l0 optimization problem, leading to accurate DOA estimation. The simulation results demonstrate that, even under conditions where the Alpha stable distribution is mixed with Gaussian colored noise and the mixed signal-to-noise ratio is as low as 0, the proposed algorithm achieves a root mean square error of 0.516 4°in DOA estimation, fully illustrating its effectiveness and high accuracy in complex noise backgrounds.