The recognition and positioning capabilities of the visual perception terminal of the fruitpicking robot system are crucial indicators to increase fruitpicking success rates in the complicated agricultural environment. A realtime multitask convolutional neural network SegYOLOv5 suited for autonomous Pitaya fruit image detection for the visual system of the picking robot was proposed in this paper using Pitaya fruit with complicated shape as the research object. The network is enhanced based on the primary architecture of YOLOv5′s convolutional neural network. The multitasking target recognition and detection task of image detection and semantic segmentation is realized, and the overall performance of the model is substantially improved, by extracting threelayer enhanced features as the input of the improved cascaded RFBNet semantic segmentation network layer. With a mean Average Precision and mean Intersection Over Union of 9310% and 8364%, respectively, for the testing dataset, the enhanced SegYOLOv5 network architecture can adapt to the boundary-sensitive image semantic segmentation agricultural scene, compared with YOLOv5s+original RFBNet and YOLOv5s+BaseNet models, it is 123% and 274% higher than the former, and 238% and 145% higher than the latter. The average detection speed of SegYOLOv5 can reach 7194 fps which is 4079 fps faster than EfficientDetD0, and the mean Average Precision is 58% higher. The center of mass of Pitaya fruit may be precisely positioned in real time as the best picking position using the endtoend output of SegYOLOv5 detection output and the fusion of image geometric moment operator. The improved algorithm has high robustness and versatility, which lays an effective practical foundation for fruit picking robot based on visual perception.