Aiming at the problem that the recurrent neural network has a single feature extraction and insufficient processing of spatial information of the feature, a two-branch fusion human behavior recognition model based on bone is proposed. The model is extracted by the two-branched network of two-way cyclic gate network and multi-scale residual network, which obtains rich feature information in time and space, and increases the attention mechanism in the bidirectional cyclic gate network to further improve the performance of the whole network, and finally the feature information is classified through the classifier to obtain action. Experiments were conducted using the UCF101 and HMDB51 datasets, respectively, with an accuracy rate of 98.0% and 67.8%, respectively. Through experimental tests, it is proved that the model can obtain more complete feature information and has good performance indicators.