Measuring the rotation angle of a worm wheel is the core basis for evaluating the dynamic performance of worm gear transmission and for early fault diagnosis, such as machining errors and assembly eccentricity. Currently, existing research mainly focuses on the static measurement of the worm wheel end face. Although high precision can be achieved by optimizing calibration and image processing algorithms, the measurement systems are structurally complex, lack the capability to evaluate transmission stability during operation, and thus have limited engineering applicability. To enhance the flexibility of the measurement system and dynamically evaluate transmission smoothness, this paper proposes a precise rotation angle measurement method considering dynamic tracking of the worm wheel side surface. First, bilateral filtering and CLAHE are applied to the image sequence over a complete rotation cycle of the worm wheel, and the region of interest is determined using the Otsu thresholding method. Second, the Canny operator is introduced to improve the oriented features from accelerated segment test (FAST) and rotated binary robust independent elementary features (BRIEF) (ORB) feature extraction algorithm, so as to obtain a stable feature point set with high response, and a multi-scale pyramidal optical flow method and Kalman filter are integrated to achieve robust tracking of feature points between adjacent frames. Finally, the rotational component is extracted by constructing an inter-frame motion model, thereby completing the calculation of the inter-frame rotation angle and cumulative rotation angle of the worm wheel, and the transmission smoothness is evaluated based on the peak-to-peak value of rotation angle fluctuation. Experimental results show that, under different illumination conditions, rotational speeds, and numbers of teeth, the mean absolute error of the inter-frame rotation angle measured by the proposed method is controlled within 0.003 9°~0.004 7°, the root mean square error is 0.005 3°~0.006 7°, the 95th percentile absolute error is 0.010 7°~0.018 2°, the cumulative rotation angle is highly consistent with the encoder results, and the maximum deviation of rotation angle fluctuation is controlled within 0.024 7°. The proposed method can effectively achieve precise measurement of the worm wheel rotation angle during worm gear transmission without requiring an end-face reference, showing strong resistance to motion interference, good dynamic detection capability, and promising engineering application prospects.