Computer-vision-based surface defect detection of metal planar materials is a research hotspot in the field of metallurgical industry. The high standard of planar surface quality in metal manufacturing industry requires that the performance of automated visual inspection system and its algorithms is constantly improved. This paper attempts to present a comprehensive survey on surface defect detection technologies based on two-dimensional and three-dimensional machine vision by reviewing over 110 publications for some typical metal planar materials products of steel-, aluminium-, copper-plate and strips. According to the nature of algorithms as well as image features, the existing 2-D methodologies are categorized into four groups: statistical, spectral, model-based and machine learning. The 3-D defect detection technologies are divided into photometric stereo, laser scanner and structured light measurement methods on the basis of the way of 3-D data acquisition. These classical algorithms and emerging methods are introduced, analyzed and compared in this review. Finally, the remaining challenges and future research trends of visual defect detection are discussed and forecasted in an abstract level.