To solve the problem of fault of followers in multiple unmanned aerial vehicles (UAVs) formation flight, an adaptive fault-tolerant control method based on observer is designed. Firstly, based on the leader-follower method, the formation model of UAVs and the fault model of followers in the formation are established, which are divided into the position subsystem and the yaw angle subsystem. Secondly, an observer is designed to observe the fault and the state in the position subsystem, and combined with the observed state and fault, a state feedback control law is constructed. Then, the control design scheme of yaw angle subsystem is given based on the adaptive method. The final bounded convergence of the tracking error of the system is proved based on the Lyapunov theory. Through simulation, compared with the method based on the robust fault estimation, the algorithm in this paper reduces the total tracking time and steady-state error of the system by 76% and 70.3% respectively, and both of them are significantly reduced compared with the traditional observer method, which proves that the algorithm in this paper can better overcome the adverse effects caused by the deviation fault and effectively realize the formation flight of quadrotor unmanned aerial vehicles.