Multi-sensor measurement technology is considered to be a very effective solution in surface metrology. Aiming at the problem of modeling and fusion of multi-scale complex data sets, this paper proposes a multi-source point cloud data fusion framework based on Gaussian process. Firstly, a robust point cloud registration method with adaptive distance is proposed to unify coordinate systems of different measurement datasets. Then, by introducing adjustment theory, the residuals between multiple independent data sets from different sensors are approximated, and a Gaussian process model based on Matern kernel function is constructed. Finally, the method is verified by simulation verification and practical application, and a series of comparative experiments with existing methods are carried out to verify the effectiveness of the method. The results show that the method can fuse multi-sensor datasets with higher fusion accuracy and faster computational efficiency.