In this paper, a fast and accurate identification of unsound kernels of wheat (black embryo, wormhole and damaged) is introduced via the convolution neural network (CNN) model. The hyperspectral images of 116 bands in the range of 493 to 1 106 nm, which includes normal kernels (484 grains), black embryo kernels (100 grains), wormhole kernels (100 grains) and damaged kernels (100 grains), are collected. We take one sample out of every five bands to construct the training sets of the 24 bands respectively, and use the proposed model to establish the identification model of unsound kernels of wheat. Experimental results indicate that, by using the proposed model, the recognition rate of black embryo, wormhole and damaged grains is maintained at above 94%, 95% and 92% respectively. We further improve the model by modifying the learning rate and the number of iterations, which end up improving the average recognition rate of black embryo, wormhole and damaged grains in each band by 0.624%、0.47% and 0.776%. We combine the hyperspectral imagery of all 24 bands to reconstruct the training set and retrain the CNN model. The total recognition rate of black embryo, wormhole and damaged grains was increased by 0.31%, 0.13% and 0.46%, respectively. For our studies, we find that the accuracy of unsound kernels of wheat grain recognition, can be effectively improved using hyperspectral data and the proposed CNN model.