The production workshop environment is complex, with numerous equipment and highly autonomous and uncertain personnel activities. Traditional manual observation methods are difficult to achieve efficient real-time control when facing massive monitoring data. To improve the automation monitoring level of workshop personnel behavior and ensure production safety, a behavior recognition algorithm based on improved DETR is proposed. Through on-site research in the smart workshop, various work behavior and abnormal behavior data were collected to construct an infrared behavior dataset for the workshop, and an improved algorithm was designed based on this. In response to the shortcomings of the original algorithm, relative position encoding is introduced and a spatial modulation joint attention mechanism is adopted to improve the network’s localization accuracy of the object to be detected in the global features. In addition, by introducing Gaussian distribution weights of the object to be detected, the network decoder can more efficiently recognize behavioral features. The experimental results show that the improved algorithm has improved recognition accuracy by 6.97% on self built datasets compared to the original algorithm, and also performs well on public datasets. This improvement method not only provides a more efficient solution for monitoring the behavior of workshop personnel, but also provides strong technical support for the automation and intelligent development of smart workshops.