To address the limitations of existing object detection algorithms in remote sensing small object detection—such as large model size, high computational complexity, and low detection accuracy—this paper proposes a lightweight remote sensing detection algorithm based on YOLO11, named EDB-YOLO11. The algorithm introduces improvements from two aspects: network architecture and loss function. First, the EMBC module is designed to replace the original C3K2 module, which effectively enhances the feature representation capability of the network. Second, a novel Downsample module is employed instead of the traditional convolution-based downsampling, which reduces the number of parameters and computational cost while improving the feature extraction ability of the backbone. Third, BiFPN is adopted to replace the original PANet for feature fusion, significantly enhancing the network’s multi-scale feature integration efficiency. Finally, a new loss function called Focal-WIoU is proposed by combining the advantages of the Focal mechanism and WIoU loss. This design enables the model to focus more on high-quality training samples and reduce the impact of low-quality samples, thereby improving overall detection accuracy. Experimental results show that EDB-YOLO11 reduces the number of parameters by 27.18% and the computational cost by 16.43%. On the VisDrone2019 dataset, the mAP@0.5 increases by 3.4%, and the mAP@0.5：0.95 improves by 1.8%. On the generalized remote sensing datasets SIMD and MAR20, the mAP@0.5 improves by 3.8% and 0.3%, respectively, while the mAP@0.5：0.95 improves by 3.0% and 0.2%, respectively. These results demonstrate the effectiveness and superiority of the proposed EDB-YOLO11 algorithm in remote sensing small object detection tasks.