To address the challenge of small and highly variable defect sizes in photovoltaic panels, an improved YOLOv8n-based model named YOLO-RMFP is proposed. First, by integrating an efficient multi-scale attention mechanism with receptive field attention, a Receptive Field Mixed Attention mechanism is introduced. This mechanism enables the model to focus on features at multiple scales while addressing the parameter-sharing limitations of conventional multi-scale attention, thereby enhancing the detection accuracy for tiny defects in photovoltaic panels. Second, the Receptive Field Mixed Attention mechanism is integrated with the Spatial Pyramid Pooling module to enhance the model’s capability to capture multi-scale features and focus on complex regions. This integration improves the model’s ability to suppress noise in complex backgrounds, thereby further boosting the detection precision of small defects in photovoltaic panels. Then, feature maps of different resolutions from the YOLOv8n backbone are fused with an improved multi-scale feature fusion pyramid network. This enhances the interaction of feature information, enabling more comprehensive feature extraction and improving overall detection performance. Finally, based on the PIOU loss function, the model adjusts the weightings of defect samples according to their detection difficulty. This improves the localization accuracy and effectively mitigates the problem of sample imbalance in photovoltaic defect detection. Results from ablation and comparative experiments show that the YOLO-RMFP model improves detection accuracy, with mAP@0.5 and mAP@0.5:0.95 increasing by 3.1% and 6.5%, respectively. Precision and recall are also enhanced by 4.2% and 3.5%, respectively. These results demonstrate that the proposed model meets the performance requirements for photovoltaic panel defect detection.