Aiming at the problem of non-commutative errors in the large dynamic environment of MEMS inertial navigation system, an improved high-order iterative attitude optimization algorithm is proposed. In order to solve the influence of non-exchangeable errors on the entire inertial navigation system in a large dynamic environment, the traditional equivalent rotation vector algorithm is deduced. For this algorithm, it only relies on increasing the number of subsamples to improve the solution accuracy, ignoring the problem that high-order terms will cause large errors in a large dynamic environment. Using the method of fast and slow loops, the rotation vector solutions of different orders are obtained respectively, and then the iterative solutions of the fast and slow loops are obtained through the periodic iterative algorithm. Finally, through the large dynamic environment simulation experiment and the high-frequency swing dynamic experiment of the high-precision three-axis turntable, the performance advantage of the higher-order iterative algorithm is verified. The experimental results show that in a large dynamic environment, compared with the traditional algorithm, the improved high-order iterative attitude optimization algorithm improves the accuracy by two orders of magnitude.