Abstract: Abstract: An accurate recognition of fruit and vegetable depends mainly on the occlusion of vine, leaf, and light during robotic harvesting at present. In this study, a feasible recognition algorithm was proposed to efficiently identify the ripe tomatoes in the natural environment using YOLO v4 and HSV. The data set of mature tomatoes was also collected to capture some obscure images with the vines and leaves or color-changing by light under the complex growth environment. Once the original YOLO v4 network was utilized to identify the tomatoes after learning these samples, some tomatoes in the green ripening and color transition stage were taken like in the mature stage. Therefore, an HSV processing was added into the detection box of the original YOLO v4 network, in order to segment the red region of tomatoes. The specific tomatoes were taken as the target output to improve the accuracy of recognition if the red areas of segmentation reached a critical proportion in the detection box. The size of the proportion presented an important impact on the accuracy of recognition. The recognition performance was also compared on the test set under different proportions. As such, the proportion of 16% was taken as the tomato recognition at the mature stage. At this time, the highest recognition accuracy of the combined YOLO v4 and HSV was 94.77%, 4.30 percentage point higher than that of the original. The detection speed of a single image in the workstation was 25.86 ms. It indicated that the addition of HSV processing was widely expected to improve the accuracy of the original network. Furthermore, the improved network was also used to effectively remove immature tomatoes that cannot be recognized by the improved Cascade RCNN. In addition, the running time was tested ranging from the calling RealSense D435i to the first target tomato on the workstation and the miniature industrial computer. It was found that the average time of recognition was 0.51 s on the workstation, and 1.48 s on the miniature industrial computer, using the combined YOLO v4 and HSV from turning on the camera to the first target detection. Consequently, the improved algorithm was fully met the real-time requirements of mechanical picking. This finding can also provide a strong theoretical basis for the accurate, efficient, and real-time recognition of fruit and vegetable picking using robots in a complex environment.