In order to solve the problem of maximum pooling loss of information and average pooling fuzzy features, improve the time-frequency image recognition efficiency of the model and reduce the model complexity, A CNN network model using a deep detachable small Convolutional kernel for down-sampling and CBAM is proposed for fault diagnosis of bearings. Firstly, in the pooling layer except the last layer, the depth separable small convolution layer is used to replace the pooling layer to realize the down-sampling function of the pooling layer. Secondly, CBAM is introduced in the last pooling layer to pay more attention to the fault features represented by time-frequency images to improve the computational efficiency of the model. Thirdly, global average pooling is used instead of traditional full connection layer to further reduce the number of model parameters. Finally, CWRU bearing vibration data and self-made experimental platform data were used to verify the validity and feasibility of the proposed method in rolling bearing fault diagnosis. Experimental results show that the fusion depth separable small convolution kernel and CBAM improved CNN model can effectively reduce the training parameters and computation required by the model, and achieve better performance in recognition accuracy.