Imagebased action recognition is still a very challenging task because it is disturbed by the differences in the background information of the images in the class and the similarity of the behavior between the classes. Some action categories are very similar in terms of human poses and facial expressions, so extracting salient features from various parts of the image that are rich in semantic information is essential to improve the accuracy of action recognition. Drawing on the advantages of the bilinear pooling model in finegrained image classification, and to avoid this model which containing a lot of background noise to affect the recognition accuracy, an improved bilinear pooling model is proposed for action recognition in the paper. The model uses channel and spatialwise attention mechanism to focus on the important targets in the image, and generates RoI by integrating multilayer attention mask, which can effectively suppress the background noise information in the image and improve the accuracy of action recognition. Our method achieves the accuracy of 8524% on the Stanford-40 dataset, and the accuracy of 8457% on the custom 60 kind of action dataset.