In order to improve the current road damage detection methods in complex environment detection difficulties, serious detail texture loss, low efficiency, a multi-scale feature fusion lightweight YOLO (MSL-YOLO) method is proposed. Firstly, based on the improvement of YOLO11n, the Feature fusion channel attention (FFCA) module is designed to improve the weight of damage information, strengthen the extraction of feature information, and reduce redundant information. In order to better capture damage targets of different sizes in complex environments, a multi-scale feature enhancement (MSFE) module is designed to enhance the multi-scale feature fusion capability of the model and further improve the detection performance. In order to realize the Lightweight model and real-time detection, lightweight network (LNet) is introduced in Neck to reduce the computational complexity of the model and facilitate the deployment and application of the model. The experimental results show that on the RDD2022 road crack dataset, the proposed method has an average detection accuracy of 52.5%, and the number of model parameters is 2.3×106, which is 1.8% higher than that of YOLO11n algorithm, and the number of parameters is 11.5% lower. It can not only meet the requirements of high precision, high speed and lightweight for road damage detection, but also has strong robustness and real-time.