Aiming at the problems of large amount of network computation and poor detection effect in the current traffic sign detection method, a lightweight traffic sign detection method with embedded coordinate attention mechanism is proposed. First, the coordinate attention mechanism CA module is embedded in the residual block of MobileNetv2 to retain the coordinate information in the channel attention; Secondly, the improved MobileNetv2 is used to lighten the YOLOv4 backbone network, and the depthwise separable convolution block is used in PANet to reduce the amount of computation; Then, ASFF adaptive feature fusion is used to improve the PANet structure to balance the inconsistency of different feature layers. Finally, attention is added to the feature fusion module to increase the weight of the target information; and the K-Means++ algorithm generates new a priori box cluster centers. Experiments show that the weight file is reduced by 60% from 136M to 54.5M, the network volume is reduced by 80%, and the accuracy reaches 96.84%, lose only 0.46% accuracy compared to YOLOv4 network.