Polarized Synthetic Aperture Radar can work in multiple polarization modes. Using multiple polarization echo data to achieve ground feature classification is an important application of polarization data processing. At present, there are still some problems in the application of convolutional neural network in the field of polarization feature classification. Including the information redundancy and dimension disaster caused by multi-dimensional polarization decomposition feature information To mitigate these problems, this paper proposes a fully convolutional network model based on feature fusion. Firstly, By designing a full convolutional network structure with two encoding layer branches, the deep features are extracted for the polarization decomposition feature and polarization scattering feature respectively to realize the separation of multi-dimensional feature information. Then, the attention feature fusion mechanism is adopted to realize the feature fusion of two branches, and the learning ability of the network is redistributed by sharing the attention weight of the connection layer learning channel. In addition, an improved Atrous Space Pyramid Pooling is introduced to improve the multi-scale prediction ability of the model. The experimental results show that the overall accuracy of polarization data sets in two different regions is 96.43% and 99.60% respectively, and the prediction time is 17.3s and 10.1s. The classification accuracy is improved without greatly increasing the prediction time, and the effectiveness of the algorithm is verified.