In recent years, anomaly detection plays a more and more important role in the analysis and utilization of vessel trajectory data, and has become a hot research direction in the field of navigation. The aims of detecting abnormal vessel trajectory are to study the behavioral characteristics of individuals or groups vessel and find traffic patterns and traffic characteristics hidden inside. The concept and classification of abnormal behavior of vessels are analyzed mainly from the aspects of ship position and behavior, the recent theoretical research progress in detecting abnormal vessel trajectory is summarized, the advantages and disadvantages of each method used are reviewed, and the problems and challenges in the detection of abnormal vessel trajectory are discussed finally.