In the application scene of human-computer interaction in the process of upper limb rehabilitation training, it is difficult for a single sensor to track the trajectory of the upper limb accurately and stably. To address this issue, a multi-mode perception fusion scheme based on the inertial sensor and the Kinect sensor is proposed. To solve the problem of cumulative error of inertial sensors in trajectory tracking, an error correction algorithm based on polynomial compensation and zero constraint is proposed. To solve the problem that the imaging quality of the depth image of the Kinect sensor affects the effect of bone tracking, a variable parameter pixel filter is proposed to repair the depth image. Through the motion experiment of inertial sensor, the effectiveness of the cumulative error correction algorithm is verified, and the X-axis displacement error is 7. 8 mm. The depth image restoration results show that the loss of depth pixels is reduced by about 50%, and the image is clearer. Experiments show that the depth image can effectively track the bone trajectory. Finally, the feasibility and effectiveness of the upper limb trajectory tracking scheme based on the multi-mode perception are verified by the trajectory data fusion experiment, and the computing delay is only 1. 2 s.