The grid-based image speeds up the implementation of the algorithm in the GMS (grid-based Motion Statistics) matching algorithm. However, the feature points at the edge of the Grid are not effectively processed, which leads to the existence of wrong matching pairs. This paper proposes an image mismatching elimination algorithm based on adaptive margin mesh motion statistics. Firstly, the adaptive algorithm is used to calculate the optimal distance of the grid edge, and the feature points of the grid edge are assigned to other adjacent grids, so that these feature points can effectively play a supporting role for the correct matching points and improve the score of the correct matching points. Finally, the statistical characteristics representing the motion smoothing constraint were used to eliminate the wrong matching points in the initial matching. Simulation experiments show that the recall rate of the proposed method is about 10% higher than that of the GMS algorithm, and the real-time performance is also about 30% higher. Compared with the SIFT algorithm, the running time is shortened by 17 times on average. Compared with SURF algorithm, the number of correct matches is increased by 8 times on average, which fully indicates that the wrong matching points can be removed effectively and efficiently, and the image matching quality can be further improved.