Arrhythmias is a common cardiovascular disease, which seriously affects the quality of life and safety of patients. The automatic classification of arrhythmia utilizing electrocardiogram (ECG) is of great significance for timely diagnosis and prevention. An arrhythmia classification method using multi-feature fusion is proposed. Firstly, the short time Fourier transform (STFT) features and wavelet transform (WT) features are respectively extracted from denoised ECG. Then, its deep STFT features is obtained by the branch aggregated residual network (BCAR-NET) with STFT features as input, and its deep WT features is obtained by the 1D-CNN with WT features as input. Moreover, the LSTM is used to extract deep ECG features. Finally, a fully connected network is used to concatenate and fuse the three deep features, then arrhythmia classification is realized. The proposed arrhythmia classification method is evaluated on the MIT-BIH arrhythmia dataset. The accuracy of the proposed method is 98.66%, and the macro-average F1 score is 94.22%, which is better than traditional arrhythmia classification methods. The experimental results show that the constructed multi-feature fusion network improves the classification performance of arrhythmia by effectively exploiting the complementarity between deep STFT features, WT features, and ECG features.