Prohibited items detection in X-ray security inspection is widely used to maintain public traffic safety and personal safety. In order to solve the problems of variable shape and scale, severe overlap and occlusion in Xray images, an improved YOLOv5s model combining deformable convolution and attention mechanism is proposed for prohibited items detection. Firstly, deformable convolution is introduced into the backbone network to enhance spatial feature information extraction by learning sampling offsets to adapt to different deformations of objects. Secondly, the mixed convolution attention module is used to enhance the model’s ability to perceive the detected target and suppress irrelevant background interference. Then a channelguided atrous space pyramid module is constructed to obtain more accurate global contextual information and improve the model′s ability to identify overlapping occlusion targets. Finally, the CARAFE operator is used to replace the nearest neighbor interpolation to make full use of the content information in the upsampling process and improve model’s detection accuracy. The experimental results on the SIXray_OD and OPIXray datasets show that the model’s mAP@05 is 21% and 18% higher than the original YOLOv5s, reaching 906% and 900%, respectively. Compared with many existing advanced algorithms, it has better detection accuracy and realtime performance.