Due to the influence of non-linear factors on the optical fiber displacement sensor, the measurement error of the sensor is relatively large. To this problem, this paper proposes a compensation measure to reduce the impact of these non-linear factors, using the firefly algorithm (FA) to optimize the back propagation neural network (BPNN) hybrid algorithm to improve the optical power value received by the sensor. The algorithm not only uses the search performance of the Firefly algorithm to find the best position of the particle population, but also utilize the strong local optimal weight threshold search performance of BPNN, and finally achieves the goal of preventing BPNN from falling into the best optimization situation in some samples. During the experiment, the optical power values received by the two parts of the sensors are exploited as data input into the FA-BP algorithm for training and optimization. Finally, compared with BPNN and particle swarm optimization BP neural network (PSO-BP), the FA-BP algorithm with higher convergence accuracy and few iteration steps can effectively improve the accuracy of sensor data and the running speed of the program.