To address challenges such as complex backgrounds, varying scales, and the difficulty of detecting small objects in X-ray security images, we propose a lightweight contraband detection algorithm named LEM-YOLO, which focuses on enhancing edge and multi-scale features. First, a Lightweight Edge Feature Enhancement module (LEFE) is designed to construct the EFE_C2f, enhancing the model"s capability to extract edge features. Next, we develop an Efficient Multi-level Feature Fusion Pyramid Network (EM-FPN) that utilizes Dynamic Upsampling (Dysample) and the Hierarchical Scale Feature Pyramid Network (HS-FPN) to enhance multi-scale feature fusion and reduce computational redundancy. Additionally, a Dynamic Feature Encoding module (DFE) is employed to preserve global information for small-sized objects. Finally, Shape-IoU is used as the bounding box regression loss function, focusing on the shape and scale of the bounding boxes to improve object localization accuracy. Experimental results on the publicly available SIXray dataset show that LEM-YOLO achieves a mean Average Precision (mAP) of 94.63%, which is a 2.56% improvement over the original algorithm. Furthermore, the model size is reduced by 50.67%, making it better suited for contraband detection scenarios compared to similar algorithms.