Aiming at the existing distracted driving detection algorithms, this paper constructs a YOLOv8-EFM distracted driving detection and recognition model based on improved YOLOv8-pose. Firstly, by replacing the backbone network of YOLOv8-pose with EfficientViT, combined with the complementarity between CNN and VIT, the detection accuracy is improved; secondly, replacing the Bottleneck module in C2f with FasterBlock module, increasing the detection rate and reducing the model parameters; finally, the lightweight MLCA attention module is added after SPPF, achieving a good balance between model size and accuracy. The experimental results show that the YOLOv8-EFM model constructed in this paper can detect mAP 50 with 98.9%, and the model size is only 9.7 M. This method can not only detect the specific distraction behavior, but also detect the human skeleton of the upper body, which can be effectively applied in the detection scene of distracted driving.