Point clouds have unique advantages due to its rich geometric information in computer vision field. Most of the existing point cloud classification and segmentation methods based on deep learning can identify the objects with canonical orientations. In real applications, there are problems of rotation transformation. In this paper, we propose a lightweight framework EFRI-N, namely, rotation invariant network of point cloud based on ellipsoid fitting, focusing on pointset rotation problems. We design a pre-network module to extract the rotation-invariant features. The ellipsoid fitting algorithm is used to identify the direction of the point clouds and obtain the rotation-invariant coordinate. Then the original features are mapped to the coordinate, and the rotation-invariant features were obtained by encoding the spatial and angular information. In order to obtain richer geometric information, multi-level feature connection is added to the network to enhance feature propagation and reuse. The classification and segmentation experiments are carried out by using the famous public datasets ModelNet40 and ShapeNet Parts. The results show that this method demonstrates better performance than state-ofthe-art methods in the task of processing rotating point cloud, and the network is improved by 1% ~ 62. 63%. Moreover, the computation amount and the number of parameters of the network have an order of magnitude advantage. It can meet the requirements of rotation invariance of point cloud in single object scenario and has good application value.