Phase-sensitive optical time domain reflectometer (Φ-OTDR) usually uses coherent detection to achieve long-distance, distributed and high-sensitivity vibration detection. To accurately obtain the position and phase information of the vibration signal, the quadrature demodulation algorithm is an important technology widely used at present, but the algorithm has the limitation of time-consuming. In order to solve this problem, a fast demodulation scheme of Rayleigh scattering signal based on field programmable gate array (FPGA) is proposed. The pipeline structure is used to realize the synchronization of sensor data acquisition and data demodulation. Two orthogonal signals are obtained by digital quadrature mixing technology. The finite impulse response low-pass filter is used to remove the high-frequency component, and the coordinate rotation digital algorithm (CORDIC) vector mode is used to realize the hardware demodulation of the vibration phase, which can improve the overall real-time performance of the coherent detection Φ-OTDR system. The experimental results show that the scheme can successfully realize the positioning and phase reduction of the vibration signal under the condition of a detection distance of 40 kilometer. When the detection distance unchanged and the pieces of data acquisition increased to 4 000, the FPGA demodulation scheme only takes 1.60 seconds, which is 145.61 seconds shorter than the traditional host computer CPU demodulation scheme, thus providing a reference for the real-time demodulation of Φ-OTDR vibration sensing data.