In the field of autonomous driving technology, the identification of pavement roughness directly influences subsequent driving decision-making processes. However, existing algorithms for pavement roughness recognition suffer from issues of low accuracy and slow recognition speed. Addressing this challenge, a Hidden Markov Model based pavement roughness recognition method is proposed, leveraging an improved multi-scale feature extraction network. Significant enhancements in both recognition accuracy and speed are achieved by an enhanced multi-scale convolutional neural network, which autonomously learns and extracts hierarchical features from raw data. Subsequently, t-SNE visualization is applied to the extracted features for improved understanding and analysis of feature distributions. Finally, a Hidden Markov Model is utilized for feature recognition. Experimental results demonstrate recognition accuracies of 99.6% for simulated data and 98.6% for real-world collected data, thereby proving effective for pavement roughness recognition.