Facial expressions can correctly reflect people's inner activities, but due to the complexity and subtlety of facial expressions, accurate recognition of facial expressions is still a big problem. This paper designs a method based on convolutional neural network (CNN) and long short term memory (LSTM), so that the computer can recognize the expression of human face The loss function uses focal loss. The framework includes three aspects: (1) two different preprocessing techniques are used to deal with the illumination change and preserve the edge information of the image. (2) The preprocessed image is input into two independent CNN layers for feature extraction. (3) The extracted features are fused with LSTM layer. We use FER2013, JAFFE and CK + data sets to verify the accuracy of the model, and select FER2013 data set to make a mixed matrix. The results show that the accuracy of our model on fer2013 data set is improved by 9.65% compared with the current advanced model, and it also performs well on Jaffe and CK + data sets. The results show that our proposed model has strong generalization ability.