Aiming at the situation that CFNet, the back propagation filter tracking algorithm based on the siamese network, is likely to cause the model drift tracking effect to decrease when it encounters the interference of similar objects or the background information is similar to the foreground target, a siamese network target tracking algorithm fused with semantic feature network is proposed. In image processing, through the deep network of deep convolutional neural network, rich semantic information can be extracted. These semantic information can cause similar interference, motion blur, severe target deformation, etc. In situations, it is very useful to identify the target. In the proposed algorithm, a semantic feature network is added to the original network structure of CFNet, which is complementary to the appearance feature network of CFNet. The training of the two feature networks is independent to maintain the heterogeneity of the two features and obtain their respective response maps. Later, the fusion is performed by calculating the confidence of the two response graphs, which improves the discriminative ability of algorithm. Tests show that, the algorithm in this paper achieves the optimum and can track the target effectively compared with other 5 commonly used algorithms.