Interactive system is the basis for the interaction between service robots and people. Using RealSense 3D camera as a sensor, a multimode interactive system with the elderly assistance robot was developed. It mainly implements user registration and verification based on face recognition, news on demand and calendar reminder based on gesture recognition and speech recognition. Experimental tests show that the accuracy of face recognition is 93%, the accuracy of gesture recognition is 72% and the accuracy of speech recognition is 90%. The interactive system has the advantages of userfriendly interaction, network independence and high recognition speed, etc. It integrates with the Omnidirectional mobile robotics platform to design a userfriendly, multifunctional and lowcost elderly assistance robot for empty nesters.