In order to solve the problem of insufficient transmission performance in satellite communication and deep space communication in the system of cascaded binary low-density parity-check (LDPC) codes and high-order continuous phase modulation (CPM), a multivariate LDPC-CPM optimization method is proposed. Firstly, cascading to form a multivariate LDPC-CPM system achieves better transmission error performance. Secondly, considering that the problem of the traditional decoding algorithm of nonbinary LDPC codes is too complex and hard to implement in hardware, an improved Mixed-Log-FFT-BP algorithm is designed. The solution of the ratio reduces the decoding complexity. Finally, in view of the problem of too many parameters in multivariate LDPC-CPM system, a parameter optimization method with gradually convergent performance is proposed. The simulation results show that, compared with the system before optimization, the optimized system has a 1~1.2 dB improvement in the bit error performance under the medium and high signal-to-noise ratio and the system complexity is lower.