Aiming at the binocular vision system of the space manipulator of the insulated bucket arm vehicle, the optimal design and calibration of the parameters of two panoramic cameras in the vision system are studied in this paper, and a model is constructed after the optimal configuration of the camera position, direction and mirror shape in the system, so as to obtain higher accuracy of 3d data measurement. In order to construct the optimal configuration of the visual system, the analytical formula of the 3D measurement error model is derived according to the error propagation analysis, the quantization accuracy of image pixels and the variation of angular resolution in the process of data calculation. Then, this formula can be used in framework optimization to find the best system configuration for different shapes of system Settings. For the general case with cuboid 3D measurement area and camera placement area, the suboptimal solution of system configuration is derived by using the proposed analytical formula, and the accuracy of the suboptimal solution is proved to be close to its optimal solution. Finally, the experimental results of simulation and practical application cases show that the proposed method can effectively improve the accuracy and robustness of binocular ranging, and the running speed is fast.