To address the current limitation in polarization image fusion technology—where the focus is predominantly on the visual quality and statistical metrics of the fused output while neglecting its applicability to subsequent high-level vision tasks—this paper proposes a dual-branch feature extraction architecture for polarization image fusion, guided by semantic segmentation. The fusion network comprises an encoder, a fusion layer, and a decoder. In the encoder, a dual-branch feature extractor—composed of GRDB and Swin Transformers—is constructed to extract local polarization features and global intensity information from the source images. Within the fusion layer, an INN is employed to model the inter-feature correlations, enabling lossless enhancement and effective fusion of the polarization characteristics. In the decoder, Restormer serves as the core building block to reconstruct and preserve high-frequency details and structural scene information from the fused features, thereby enhancing image clarity and generating the final fused result. To enrich the fused output with task-relevant semantics, the fusion network is cascaded with a segmentation network during training. The semantic segmentation loss is leveraged to guide the backpropagation of high-level semantic information, thereby optimizing the fusion network and improving the utility of the fused images for advanced vision tasks. Experimental results demonstrate that the proposed network achieves superior performance in both subjective visual assessment and downstream semantic segmentation tasks. Moreover, it outperforms existing fusion methods in objective metrics, with notable improvements of 27% in EN and 16.8% in SSIM.