When detecting dim and small targets with low signal-to-noise ratio in infrared image sequences of sky background, the simple traditional algorithm has some problems, such as complex preprocessing process, difficult feature design, difficult to determine control parameters, low detection accuracy and so on. Through the introduction of deep learning technology, a combined algorithm is proposed, which can significantly improve the detection effect of the algorithm. In the infrared image sequence, firstly, the moving target is detected with high accuracy by using the spatio-temporal feature extraction network based on YOLOv3 in the starting frame, and then the traditional method based on local contrast feature is used to detect the target quickly in the subsequent frames according to the speed and luminance characteristics of the target. In the infrared image sequence test data set of the sky background, the combined method achieves higher accuracy and recall than the existing methods, and the computing time also meets the real-time requirements. The results show that the two methods cooperate with each other and achieve a good balance in real-time and accuracy.