Objective The two-dimensional angular displacement table often consists of two two-dimensional angular displacement table. It is difficult to avoid installation errors when mounting a biaxial tilt sensor on a two-dimensional angular displacement table, which can lead to inaccuracies in the measurement of two-dimensional tilt angles. Meanwhile, there is a two-dimensional angular coupling between the rotating axes of the off-axis angular displacement table. And the tilt of the bottom angular displacement table will cause the angle measurement deviation of the upper angular displacement table. Based on the existing angle accuracy of the angular displacement table and the tilt sensor, we hope that a method can be proposed to correct the above linear and nonlinear angle errors through the system model establishment and error analysis. Methods Our work consists of the following parts. First, the systematic angle errors of the two-dimensional off-axis tilting system are analyzed and modeled in this paper, which are categorized into quasi-linear errors and nonlinear errors. The quasi-linear errors include vertical and horizontal tilt sensor installation errors. The nonlinear error is only caused by the off-axis layout of the angular displacement table. Second, a correction method combining the linear homography matrix and the nonlinear off-axis inclination model is proposed for compensating the two types of errors. Furthermore, the automatic correction process is designed. Finally, the calibration accuracy of the proposed method was verified by experiments. Results and Discussions In order to verify the calibration accuracy of the proposed method, the off-axis angle control experiment was carried out in the range of ±12° to verify the calibration accuracy of the proposed method. The experimental results proved that the measurement accuracy of the two-dimensional angular displacement table tilt angle is improved from 0.559° and -0.216° before correction to 0.025° and 0.013°, which reduces the error by an order of magnitude. The method can meet the requirements of precise measurement and control of off-axis tilt angles. Furthermore, in order to verify the effectiveness of the proposed method, a comparative experiment was conducted with two existing methods. The experimental results show that the proposed method is superior to the other two methods in terms of accuracy and ease of operation. Conclusions The errors of the biaxial tilt sensor mounted on a two-dimensional angular displacement table are analyzed and modeled, which are classified into quasi-linear errors and nonlinear errors. The corresponding correction method has been established by combining linear homography matrix and nonlinear off-axis inclination model. The experimental results proved that the proposed method effectively improve the measurement accuracy of the two-dimensional tilt angles. Compared with two existing methods, the accuracy of proposed method is higher, while the operation is much simpler.