FPGA is a programmable logic controller, which not only has the operating speed and stability of hardware circuits, but also has the programmability and flexibility of software. FPGA can easily perform multichannel highspeed parallel fast Fourier transform. Therefore, FPGA becomes more and more popular in online condition monitoring, vibration signal processing and edge computing. In order to make full use of the computation power of FPGA, as well as improve the accuracy in signal processing, this paper uses a rotor system and FPGAbased vibration monitoring device to experimentally study the key steps that affect the accuracy of the vibration signal processing, in order to find the optimal parameters for filtering, digital integration, and peak detection. The results show that the integration cutoff frequency greatly affects integrated velocity. The cutoff frequency should be set reasonably according to the speed of the equipment to improve the acquired RMS value. Implementing a bandpass 4th order Butterworth filter can eliminate the influence of high frequency noise and improve the stability of vibration peaks. Calculated peaks from an analog capacitor charge and discharge algorithm is stable, which is suitable for calculating the original signal peaks with “burrs”. It can be used for vibration protection. The onesecond peaktopeak value is very sensitive, which is suitable for early warning/fault detection.