Aiming the serious distortion problem caused by the measurement error of measurement while drilling (MWD) magnetometer, an online calibration method of MWD magnetometer based on federated recursion extended Kalman filter (FREKF) is proposed. Firstly, the federated fusion framework of drilling tool geomagnetic information is established. Based on the error calibration model of magnetometer while drilling, the nonlinear measurement equation is constructed as the main filter, and the sub-filter 1 is formed with the observed geomagnetic vector model of accelerometer and the sub-filter 2 is formed with the recurrent geomagnetic vector model of gyroscope. Secondly, during the sub-filter’s measurement update, we use state estimation results and its covariance matrix as one-step prediction results and covariance matrix to improve state estimation accuracy. Then the subsystem of cyclic filtering is federated to realize the high precision calibration of magnetometer. Finally, the simulation experiment and real drilling experiment are designed. The experimental results show that the FREKF can effectively calibrate the magnetometer error while drilling. Following calibration, the azimuth mean error can be controlled within 0.8°.