In order to improve the real-time performance of the activity recognition algorithm and be suitable for embedded devices with limited resources, a hardware acceleration method of the activity recognition algorithm was proposed and implemented on the FPGA platform. Traditional wearable sensor-based behavior recognition algorithms require strictly labeled data for training and classification, but the labeling process of sensor sequences consumes a lot of manpower and computing resources. To solve this problem, an attention mechanism is introduced into the traditional convolutional neural network model. , for action recognition based on weakly labeled data. Computational modules such as convolution, pooling, and attention mechanisms in the algorithm use a high-level comprehensive design. According to the operation characteristics of the model, the operation speed is improved by pipeline constraints, multi-pixel and multi-channel parallelization, and data fixed-pointization. Experiments are carried out on the Ultra96_V2 platform, and the experimental results show that the designed behavior recognition system has a recognition accuracy of 90% and a computing speed of 25.89frams/s, which is 54.15 times faster than that of a single-core ARM_A53 processor. The average power consumption of the system is 2.204W and the power efficiency is 11.75frams/s, which meets the design requirements of low power consumption and low delay.