In view of the current image restoration algorithms mainly use the R, G, B information of the image to obtain the optimal matching block, which ignoring the structural features of the image and resulting in the problem of texture discontinuity and block phenomenon in the repaired image, this paper designs an image restoration algorithm with variance adjustment strategy coupling structural features. Firstly, the gradient modulus of image is used to construct the structure measurement factor to measure the structure characteristics of image. The priority function is constructed by combining the confidence term, structure measure factor and data term to find the priority repair block. Then, the variance feature of the image is used to establish the variance adjustment strategy, considering the dynamic changes of the image texture, to find the most consistent sample block size with the current texture situation. Finally, the structure measure factor is introduced into the search process of the optimal matching block to make up for the lack of the ignored image structure features when searching the optimal matching block through R, G, B information, accurately obtain the optimal matching block, and achieve the repair of the damaged area. The experimental results show that the algorithm has better texture continuity and visual effect than the current algorithm, and the performance is better.