To ensure and improve the field measurement accuracy of the rotary table angular measurement system, a research on the separation and compensation method of rotary table indexing error based on Fourier transformation is proposed. Based on the proof of principle of Fourier transformation to separate the indexing error of the rotary table, the function model between indexing error of the rotary table and the measurement value of the reading head is formulated. According to the nature of transfer function in Fourier transformation, the relationship between the installation angle interval of the double reading head and the harmonic order of the measurement error is highlighted. The arrangement of the double reading head is optimized. The measurement value of multiple reading heads is realized on the field programmable gate array (FPGA) circuit platform. The simultaneous acquisition of the measured values of multiple reading heads is realized in the FPGA circuit platform, and the harmonic error function is calculated in real time by using the coordinate rotation digital computer. The experimental results show that the indexing error can be separated and compensated by using the optimized double reading head signals, the peak value of indexing error of rotary table is reduced from 57. 58″ to 3. 36″, and the expanded measurement uncertainty of angle measuring system of compensated rotary table is 0. 9″(k = 2).