Aiming at the problems of significant differences in polyp size, complex intestinal environment, and limited medical diagnostic equipment resources affecting detection accuracy in polyp detection tasks, a lightweight polyp detection model based on RT-DETR improvement was proposed. Firstly, FasterNet is used as the backbone network of the RT-DETR model to reconstruct the FasterNet Block module to divert redundant features while increasing attention to polyps. Secondly, the new module is designed to introduce HiLo high and low frequency separation mechanism into the attention-based intrascale feature interaction (AIFI) to separate local high frequency details and low frequency global structures, and focus on key lesions in complex backgrounds. Finally, an SBA-FPN recalibration feature fusion network is designed to replace the cross-scale feature fusion module (CCFM) to promote two-way fusion between features with different resolutions and improve the multi-scale feature fusion effect. The experimental results show that compared with the original RT-DETR model, the mAP@0.5 and mAP@0.5:0.95 values of the improved model are increased by 2.3% and 3.0% respectively, and the amount of parameters and calculations is reduced by 44.4% and 48.6% respectively. On the Br35H brain tumor dataset, the mAP@0.5 of the improved model increased by 1.3%. It can be seen that the improved model not only meets the needs of automatic polyp detection, but also meets the high-precision detection of generalized lesions in medical scenarios.