In order to effectively monitor whether the elderly fall, an external contour features extraction combining background subtraction based on human body boundary was proposed, which identify human movement behavior. Firstly, the background subtraction method was used to extract the moving objects from the video, and then the pictures of extracted moving objects were preprocessed. Secondly, the detection method of the minimum external rectangle and the center of gravity was used to extract the moving objects’ features, so as to obtain the overall external contour and the center of gravity position of the elderly. Finally, according to the different positions of human body, the movement model was established to effectively identify the movement of the monitored object, such as walking, falling, etc. The experimental results show that the algorithm proposed in this paper can effectively process the actual video, and the accuracy of human behavior recognition reaches 94. 3%.