In order to reduce the incidence of accidents caused by fatigue driving, a method is proposed to build a fatigue driving detection model by integrating convolutional neural network with face feature points and fatigue indicators. Firstly, the driver's eyes and mouth areas are located by the SSD network, and the VGG16 network learns the fatigue features contained in the eye and mouth areas. At the same time, 68 feature points of face, eye aspect ratio and mouth aspect ratio are combined to determine the driving fatigue state. Finally, the mean average precision of SSD algorithm and Faster-RCNN algorithm is calculated under the same test set. The model is applied to YawDD dataset. And the feasibility of this model is verified by simulating driving environment. The experimental results show that SSD algorithm is better than Faster-RCNN algorithm, the detection accuracy of this model on YawDD dataset is about 97.2%, and the camera can also detect the driver's state in real-time. The model is effective in detecting fatigue state and reducing the accident rate caused by fatigue driving to a certain extent.