As a common defect type, surface rust of metal fittings in transmission lines is one of the important hidden dangers endangering the safe operation of transmission lines. How to quickly and accurately discover and repair rusted metal fittings is an urgent problem to be solved in the work of transmission line inspection. This article reviews the research progress of vision-based rust defect detection methods for metal fittings in the last ten years. Firstly, the rust defect detection process of metal fittings based on traditional image processing is introduced. Then, the rust defect detection of metal fittings is summarized according to traditional image processing and deep learning methods. The application of object detection and semantic segmentation algorithms in rust defect detection of metal fittings is emphasized. Next, the self-built data sets for metal fittings′ rust defect detection and performance evaluation indexes are introduced. Finally, the existing problems of rust defect detection methods based on deep learning are pointed out and future research work is prospected.