The low signal-to-noise ratio is a major challenge for ultrasonic inspection of coarse crystal structures, which is caused by the large amount of scattering noise generated by coarse grains. To overcome the insufficient of the traditional signal denoising method in waveform distortion and amplitude attenuation, an overlapping group sparse variational method based on the non-convex penalty function is proposed for ultrasonic signals denoising. Based on the typical ultrasonic signal containing scattered noise, the influence of the main parameters of the sparse variational method for overlapping groups of nonconvex variables ( such as the type of function of nonconvex variables, regularization parameters and multiplication factors) on its noise suppression effectiveness is analyzed, and the basis for determination of suitable parameters in processing of ultrasonic signals is determined. On this basis, the overlapping group sparse variational method is applied to the noise suppression of ultrasonic signals detected from steel ingot. Results show that the method can effectively suppress the scattering noise in ultrasonic signals detected from steel ingot and improve the signal-to-noise ratio of ultrasonic imaging by more than 6 dB. The research work is a useful exploration for ultrasonic detection of coarse crystalline materials.