In view of the existing algorithms, traditional video emotion-based facial expression recognition method only pays attention to the spatial features of a single video frame, and ignores the hidden temporal features between adjacent frames. Therefore, this paper proposes a novel method to extract features using edge detection and improved recurrent neural network. Gradient edge detection can extract texture information of video frame in a more accurate way,at the same time, a kind of overlapping LSTM structure is proposed, and the recurrent neural network can acquire the hidden spatio-temporal information from the input frames. The experiments in this paper are carried out on the CK + and MMI video database. the result of 88. 4% and 69. 7% are obtained in the OCNN-RNN network respectively, and the outcome of 89. 8% and 73. 6% are acquired in the GCNN-RNN network from each database. and finally the random search is used to weight the fusion of the results of the GCNN-RNN network and the OCNN-RNN network. After the two networks are finally merged, the average recognition rate of the integrated model is 94. 6% and 79. 9% respectively, and the accuracy is better than other algorithms, the effectiveness of the proposed algorithm is proved.