In the context of measurement while drilling (MWD), there is a significant issue related to the accuracy of magnetic azimuth calculations due to substantial errors in the measurement of geomagnetic data using micro-electro-mechanical systems ( MEMS ) magnetometers,improved skill optimization (ISO) error parameter estimation method for magnetometer was proposed. Firstly, a multiparameter error model is established according to the output characteristics of the magnetometer, and according to the modulus relationship between the local geomagnetic vector and the real output vector of the magnetometer, the nonlinear error objective function is constructed by the principle of minimization, and the dot product value between the gravity vector and the magnetic field vector is a fixed value as the constraint function, and the SO algorithm is used for optimization. Due to the difficulty of estimating the magnetic error sources, the ISO algorithm is proposed on the basis of SO, and the Tent chaotic inverse learning initialization strategy is used to improve the randomness of the initial population, retain the optimal solution and increase the diversity of the solution space of the magnetic error parameters. The adaptive skill intensity factor is introduced, and the skill cross-avoidance local optimal between members is increased, the skill step size of the ISO magnetic error parameter optimization is improved, the algorithm running time is reduced, the global search ability is optimized, and the error compensation accuracy of the magnetometer is improved. Finally, the compensation performance of the ISO method is verified by turntable experiments and simulated drilling experiments, and the experimental results show that the proposed algorithm has a significant optimization effect on the error parameters of the magnetometer, and the error range of the calculated geomagnetic modulus is reduced to ±0. 2 μT, and the mean absolute error of azimuth is reduced to 2. 1°. It is shown that the output error of the magnetometer is significantly reduced after parameter optimization, and the method can effectively improve the accuracy of MEMS magnetometer measurement, obtain reliable azimuth angles, and verify the effectiveness of ISO.