Lane detection is a significant research subject in the field of intelligent driving. However, there will always be inaccurate lane segmentation and insufficient realtime processing capabilities in practical applications. Accordingly, an improved algorithm based on the Pyramid Scene Parsing Network is proposed. A main network PSPNet is built on a basis of the encoding structure, and the encoder backbone network is replaced by the lightweight MobileNet v2 network, which effectively cut down the parameter amount and computational complexity of the whole network. Hole convolution is added into the network and feature fusion is realized between different layers, which expands the model receptive field and enriches local feature information. Finally, an adaptive line fitting algorithm is used to fit different lane lines in order to obtain the final prediction result. The Caltech lane data set is come into use for testing. The experimental results show that the improved algorithm has better segmentation for different types of lane lines. Compared with the original algorithm, the Pixel Accuracy and the Intersection over Union is improved by 3.91%, 4.14%, and FPS up to 28 frames per second. The segmentation accuracy and inference speed of the proposed algorithm are superior to other comparison algorithms.