Aiming at the challenges of large differences in the morphology of multi-category defects in fabric defect detection, the dense distribution of tiny defective targets, and the demand for real-time high-precision detection, FDRT-DETR: a multi-category cloth defect detection algorithm is proposed. Firstly, the inverted residual mobile block (iRMB) module is applied in the backbone network to strengthen multi-scale feature extraction and reduce computational redundancy. Secondly, the token statistics self attention (TSSA) mechanism is invoked to enhance the model’s accuracy in capturing features in defective regions and reduce the interference of background texture. Further, the parallel atrous convolution attention pyramid network (PACAPN) is designed in the neck network to significantly improve the preservation and recognition of small target features. Lastly, for the problem of low matching quality due to the dense distribution of tiny imperfections, a matchability aware loss function (MAL) is introduced to improve the detection performance. Experimental results on Alibaba Tianchi’s fabric defect dataset show that the improved model increases the mAP@0.5 for fabric defect detection by 3.7%, achieving a speed of 64.4 frames/s with only 31.4×106 parameters, thus satisfying the practical needs of industrial production.