In order to realize unsupervised detection of wafer surface defects, an unsupervised wafer surface defect detection model with improved generative adversarial network was proposed. The model detected the defects by the difference between the target image and the reconstructed image. In this method, an encoder-decoder convolutional neural network with two layers of skip connections and memory module was used to build the generator. The skip connections were used to capture multi-scale input image features, and the memory module was used to constrain latent characteristics to enlarge the distance between real defect samples and reconstructed samples. The method also makes the model lightweight by improving the discriminator network structure. Experimental results show that the proposed model can accurately distinguish the defective wafer samples, and the area value under the ROC curve reaches 0.934, which is better than the existing unsupervised learning detection methods, and the parameters and flops of the discriminator network is reduced to less than 1 M and 60 M.