Aiming at the complicated problem of multi-parameter and mult-objective cooperative optimization of motor, a layered iterative optimization method based on nondominated sorting genetic algorithm is proposed. Firstly, the design flow and working principle of stator segment mixed excitation switched reluctance motor are introduced. Secondly, the parameters to be optimized and the optimization target of the motor are selected. After Pearson correlation coefficient is introduced to analyze the correlation between the motor parameters and the optimization target, the optimization parameters are stratified according to the correlation results. The nonlinear model of each layer optimization parameter and optimization objective is established, and the nonlinear objective model is introduced into the multi-objective optimization algorithm. Finally, the optimal individual is selected in Pareto front, the hierarchical iterative optimization of motor structure parameters and control parameters is completed, the optimal structure parameters and control parameters of the motor are determined, and the finite element analysis software is used to verify. Compared with the initial model, the efficiency of the optimized motor is slightly improved, the average torque is increased by 12.44% and the torque ripple is reduced by 64.96%. The experimental prototype is manufactured according to the optimal parameters, and the experimental results verify the effectiveness and superiority of the optimal design.