At present, the sit-up test in many school physical fitness test projects still needs to be manually counted, which not only consumes manpower, but also is inefficient. In order to promote the intelligent development of physical fitness, this paper proposes a method for counting sit-up behavior based on a fusion implementation of the Fast-Openpose human posture estimation model and Support Vector Machine (SVM). The position information of the key points of the human body in the continuous video stream of sit-ups is detected by Openpose, and the SVM is used to classify the motion features of the coordinate data of each frame of the key points of the human body. In view of the shortcomings of the original Openpose network, such as high complexity, large number of model parameters and long detection time, this paper uses FasterNet to lightweightly improve its main feature extraction part, and optimizes the more efficient single branch network structure and convolution type in the prediction branch. Finally, Spatial Group-wise Enhance (SGE) is introduced to make up for the loss of accuracy. Based on the CoCo2017 dataset, an additional 1000 image data of sit-up scene are expanded for model training, and the experimental results show that the improved Fast-Openpose reduces the number of model parameters by nearly 80% and improves the speed of keypoint detection by 110%, while losing part of the accuracy but not affecting the sit-up pose estimation. Compared with other improved models in the same series, it has more lightweight and speed advantages while maintaining similar mAP values.