When testing the analog-to-digital converters (ADCs), coherent sampling is required. However, it is difficult to achieve, so spectrum leakage will occur. In order to suppress leakage, the most common method used is windowing. The method is so strict that it is not suitable for the current high-precision ADC test, and the user should be very familiar with the window function. In this paper, a method for selecting the appropriate window function type and the span is proposed. Through the proposed method, accurate test results can still be obtained when the first side lobes of the window function are higher than the noise floor and the application range of common window functions is expanded. This makes common window functions, like Hanning window, can also be applied to 18-bit high-precision ADC test work. The simulation results show that the proposed algorithm can still obtain accurate test results over any level of noncoherency, and the maximum error of each parameter is less than 1 dB. The experimental results of a commercial 18-bit ADC are used to verify the feasibility of the proposed algorithm.