Abstract:To address the coupling of residual errors of single-visual geometric localization are coupled with the unknown physical properties of objects in unstructured environments, as well as the lack of low-level dynamic closed-loop compensation in existing visuo-tactile systems leading to uncontrolled contact forces, a visuo-tactile collaborative robotic grasping method based on visual prior guidance and tactile feedback correction is proposed. First, the GraspNet backbone network is replaced by ResUNet, and a center-of-mass constraint mechanism based on the Euclidean distance between the grasp point and the object′s centroid is introduced. This enhances the completeness of point cloud feature extraction, thereby improving the physical stability and accuracy of grasp pose detection. Second, a convolutional neural network (CNN)-Transformer stiffness estimator is trained using a tactile dataset. The online-identified soft, medium, and hard stiffness levels are input into the controller according to the admittance parameter conversion rules mapped by classification probabilities and impedance coefficients. This realizes the dynamic adjustment of grasping contact force, enhances interaction compliance, and reduces the risk of object damage. Finally, based on the deep deterministic policy gradient (DDPG), the rapidly-exploring random tree (RRT) algorithm and hindsight experience replay (HER) mechanism are integrated to construct the RRT-guided hindsight experience replay deep deterministic policy gradient (RHER-DDPG) algorithm, which fuses path guidance and experience reconstruction. By utilizing RRT expert trajectories to narrow the initial action exploration space and combining HER to relabel failed targets as successful samples, the convergence process of the Actor network mapping from high-dimensional states to actions is jointly accelerated, generating efficient grasping strategies. Comparative experiments show that, compared with the original GraspNet, the grasp detection accuracy is improved by 12.8%. Compared with the traditional DDPG algorithm, the model convergence iterations are reduced by approximately 30%. In real-world robotic grasping tests, the success rate reaches 90.3% and the object damage rate is only 7.1%, verifying the effectiveness of the proposed method.