Relative localization is a prerequisite for multiple robots in unknown environments to accomplish collaborative tasks such as formation, exploration, and rescue. A relative localization method based on ultra-wideband (UWB) bearing is proposed for positioning between robots in unknown infrastructure-free environments where satellite signals are blocked. The proposed method uses a sliding window to intercept the inter-robot bearing observations and motion trajectories over a period of time, construct the bearing cost function, and estimate the relative pose between the robots by minimizing the cost function. However, the non-convexity of the function leads traditional optimization algorithms to fall into local optimal solutions. Therefore, sparrow search algorithm (SSA) is used to optimize the cost function for the relative localization between robots. To reduce the effect of UWB bearing measurement errors, the SSA-estimated pose and odometry information are fused by a back-end pose graph optimization algorithm to achieve more accurate relative positioning. The experimental results show that the method is able to achieve an average translation error of 0.32 m and an average rotation error of 2.1° in an indoor environment with a size of 12 m×6 m.