Urban rail transit track condition monitoring is one of the critical tasks for ensuring the safety of railway transportation systems. The urban rail transit track includes key components such as rails, fasteners, bolts, and sleepers. In response to the demand for real-time and refined detection, this study, building on previous work, further investigates and proposes an innovative intelligent method based on instance segmentation for the rapid and refined identification of multiple key components of urban rail transit tracks, analyzes, and quantifies the detection results of common defects. Specifically, this research, based on the existing RTLSeg model, integrates field-of-view enhancement and image post-processing techniques, proposing an improved track image segmentation and evaluation model (ABI-RTLSeg). Firstly, to enhance the model’s learning of high-level semantic information, this study introduces a dilated spatial pyramid pooling (ASPP) module into the deep backbone network. Secondly, a convolution-based bilinear interpolation structure is incorporated into the Coord-Protonet to obtain higher-quality prototype masks and semantic information awareness. Lastly, based on the visual features of defect segmentation masks, a segmentation result analysis module is constructed, employing ellipse fitting and morphological analysis methods to analyze the safety status of common defects. Experimental results demonstrate that this method is feasible for rapid and refined detection, segmentation, and analysis of multiple target key components and common defects of railway track lines, and its performance surpasses that of the comparative baseline models. In particular, ABI-RTLSeg is able to achieve 90.91% bbox mAP and 91.67% mask mAP with the customized dataset. Meanwhile, the average inference speed reaches 25.62 fps. The average detection accuracy and recall are 100% and 99.85%, respectively. Furthermore, the feasibility of the proposed methods for assessing the severity of fastener damage and estimating key parameters of rail corrugation has been explored through multiple case studies. In summary, this study provides a new technical approach for the intelligent monitoring of rail transit track lines, which is of great significance for improving the safety and reliability of the railway transportation system.