Nonlinear Vector Network Analyzer (NVNA) is a measurement setup for characterizing the nonlinear behavior of RF/Microwave devices. It uses a phase reference to achieve the phase spectrum measurement of a complex test target, and is mainly used for the power amplifier characterization and modeling at present. This paper presents an improved twophasereference NVNA setup, and investigates its performance for mixer characterization with nonlinear behaviors. This novel NVNA setup uses a “stepped” multisine as the primary phase reference, to achieve multiband measurement on a dense spectral grid, and uses a second “auxiliary” phase reference on a coarse grid to ensure the synchronization between multiband phase measurements. As a result, the signaltonoise ratio of the test bench can be largely improved, and a complex test target with multiband and “modulated multiharmonic” components can be characterized. According to the experimental results, the proposed method can guarantee a phase measurement stability of less than ±0.2° fluctuation, and achieve the phase and harmonic distortion analysis of a mixer under complex modulation.