It is difficult to estimate the probability distribution and uncertainty of measurement results when the process of evaluating the uncertainty of coaxiality error measurement is complex and the error distribution is unknown. Based on a small amount of existing data, the probability density of the measured data is calculated by the support vector machine method (SVM), and the numerical integration of the obtained probability density distribution is carried out to calculate its estimation and standard uncertainty. The experimental results show that under the condition of small sample data, the best estimation and the accuracy of variance of the samples obtained by SVM are proved. Finally, the coaxiality error of the machine tool core shaft is taken as the experimental object, and the mentioned results are used as the experimental object. Methods to calculate the measurement uncertainty and the measurement uncertainty representation guide (GUM), Monte Carlo method (MCM) calculation results are compared to verify the simplicity, reliability and accuracy of the method.