Defect detection on outer packaging of wine bottles is of importance. A improved YOLOv3 algorithm is proposed to deal with that problem. The final result shows the improved YOLOv3 meets the production line’s requirements for accuracy and speed well. First, The SENet Module is introduced into YOLOv3 backbone network’s residual block, applying the attention mechanism to enhance the feature extraction. Second, The Adaptive Feature Fusion Network (ASFF) is introduced into the Feature Pyramid Network to fuse the feature information in different scales, which enhance the predictive ability of the model. Third, The Focus Loss function is used to solve the problem of unbalanced positive and negative samples, which will help accelerate the convergence speed of the loss function. The improved YOLOv3-ASFL achieves a mAP up to 92.33% in the self-made wine bottle cap dataset, which is 6.59% higher than the original YOLOv3, and the single image detection time is only 0.085s. The improved YOLOv3 model has a better performance and meets the needs of the wine bottle packaging production line for defect detection.