In view of the current vehicle tracking research algorithms, the Kernel correlation filtering algorithm ( KCF) has the shortcomings of single feature extraction and the inability to adapt the scale in a complex background, this paper proposes a multi-feature fusion scale adaptive algorithm. The algorithm uses color histogram information as color features, takes high-level convolution features with more semantic information and low-level convolution features with higher resolution as depth features, and performs adaptive feature fusion with color features. Then, the context image is used to constrain and optimize the target background information, and the response confidence is measured by the average peak correlation energy detection, and finally the high-confidence tracking result is used to avoid the problem of the model being vulnerable to interference. In addition, in order to achieve high target tracking accuracy, the algorithm in this paper uses a hierarchical model update strategy to update the extracted features. Experiments on the OTB100 data set show that the accuracy of the algorithm in this paper is better than other mainstream tracking algorithms Staple, SAMF, KCF, TLD, DSST and CSK are 4. 9%, 5. 7%, 10. 2%, 10. 3%, 23. 4%, 29. 7% higher.