The detection method of longitudinal tear of single signal conveyor belt based on image or sound is often affected by light and noise. In order to overcome this limitation, a damage detection method based on multi scale feature cross fusion is proposed. Firstly, Log-Mel feature extraction algorithm is used to upgrade one-dimensional sound signal to two-dimensional spectrum; Secondly, a dual input neural network is built to extract the features of the image and the sound spectrogram at the same time, and they are cross fused at different scales; Finally, according to the fused features, the damage classification is determined by the loss function. Through experiments, the detection accuracy, sensitivity of scratch and longitudinal tear of this method can reach 97.37%, 96.53% and 98.67% respectively, which is 7.04%, 6.96% and 6.4% higher than the existing audio-visual decision level fusion method. Therefore, this method can better meet the reliability requirements of conveyor belt damage detection.