Skip to main content
SpringerLink
Log in

Fruit detection in natural environment using partial shape matching and probabilistic Hough transform

  • Published:
Precision Agriculture Aims and scope Submit manuscript

Abstract

This paper proposes a novel technique for fruit detection in natural environments which is applicable in automatic harvesting robots, yield estimation systems and quality monitoring systems. As most color-based techniques are highly sensitive to illumination changes and low contrasts between fruits and leaves, the proposed technique, conversely, is based on contour information. Firstly, a discriminative shape descriptor is derived to represent geometrical properties of arbitrary fragment, and applied to a bidirectional partial shape matching to detect sub-fragments of interest that match parts of a reference contour. Then, a novel probabilistic Hough transform is developed to aggregate these sub-fragments for obtaining fruit candidates. Finally, all fruit candidates are verified by a support vector machine classifier trained on color and texture features. Citrus, tomato, pumpkin, bitter gourd, towel gourd and mango datasets were provided. Experiments on these datasets demonstrated that the proposed approach was competitive for detecting most type of fruits, such as green, orange, circular and non-circular, in natural environments.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

References

  • Ahonen, T., Matas, J., He, C., & Pietikainen, M. (2009). Rotation invariant image description with local binary pattern histogram fourier features. In Proceedings of the 16th Scandinavian Conference on Image Analysis (pp. 61–70). Heidelberg, Germany: Springer.

  • Arbelaez, P., Maire, M., Fowlkes, C., & Malik, J. (2011). Contour detection and hierarchical image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence,33(5), 898–916.

    Article  PubMed  Google Scholar 

  • Bargoti, S., & Underwood, J. P. (2017). Image segmentation for fruit detection and yield estimation in apple orchards. Journal of Field Robotics,34(6), 1039–1060.

    Article  Google Scholar 

  • Barnea, E., Mairon, R., & Ben-Shahar, O. (2016). Colour-agnostic shape-based 3d fruit detection for crop harvesting robots. Biosystems Engineering,146, 57–70.

    Article  Google Scholar 

  • Bulò, S. R., & Pelillo, M. (2013). A game-theoretic approach to hypergraph clustering. IEEE Transactions on Pattern Analysis and Machine Intelligence,35(6), 1312–1327.

    Article  Google Scholar 

  • Chakraborty, B., Gonzàlez, J., & Roca, F. X. (2013). Large scale continuous visual event recognition using max-margin Hough transformation framework. Computer Vision and Image Understanding,117(10), 1356–1368.

    Article  Google Scholar 

  • Dalal, N., & Triggs, B. (2005). Histograms of oriented gradients for human detection. In Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (pp. 886–893). Los Alamitos, USA: IEEE Computer Society Press.

  • Donoser, M., Riemenschneider, H., & Bischof, H. (2009). Efficient partial shape matching of outer contours. In Proceedings of the 9th Asian Conference on Computer Vision (pp. 281–292). Heidelberg, Germany: Springer.

  • Everingham, M., Eslami, S. M. A., Van Gool, L., Williams, C. K. I., Winn, J., & Zisserman, A. (2015). The Pascal Visual Object Classes challenge: A retrospective. International Journal of Computer Vision,111(1), 98–136.

    Article  Google Scholar 

  • Fan, H., Yang, C., & Tang, Y. (2015). Object detection based on scale-invariant partial shape matching. Machine Vision and Applications,26(6), 711–721.

    Article  Google Scholar 

  • Fergus, R., Perona, P., & Zisserman, A. (2003). Object class recognition by unsupervised scale-invariant learning. In Proceedings of the 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (pp. 264–271). Los Alamitos, USA: IEEE Computer Society Press.

  • Font, D., Pallejà, T., Tresanchez, M., Teixidó, M., Martinez, D., Moreno, J., et al. (2014). Counting red grapes in vineyards by detecting specular spherical reflection peaks in RGB images obtained at night with artificial illumination. Computers & Electronics in Agriculture,108(C), 105–111.

    Article  Google Scholar 

  • Gongal, A., Amatya, S., Karkee, M., Zhang, Q., & Lewis, K. (2015). Sensors and systems for fruit detection and localization: A review. Computers & Electronics in Agriculture,116(C), 8–19.

    Article  Google Scholar 

  • He, A. X., & Yung, N. H. C. (2008). Corner detector based on global and local curvature properties. Optical Engineering,47(5), 057008.

    Article  Google Scholar 

  • Herold, B., Truppel, I., Zude, M., & Geyer, M. (2005). Spectral measurements on ‘elstar’ apples during fruit development on the tree. Biosystems Engineering,91(2), 173–182.

    Article  Google Scholar 

  • Ji, W., Zhao, D., Cheng, F., Xu, B., Zhang, Y., & Wang, J. (2012). Automatic recognition vision system guided for apple harvesting robot. Computers & Electrical Engineering,38(5), 1186–1195.

    Article  Google Scholar 

  • Kapach, K., Barnea, E., Mairon, R., Edan, Y., & Ben-Shahar, O. (2012). Computer vision for fruit harvesting robots—State of the art and challenges ahead. International Journal of Computational Vision & Robotics,3(1/2), 4–34.

    Article  Google Scholar 

  • Kovesi, P. D. (2000). Matlab and octave functions for computer vision and image processing. http://www.peterkovesi.com/matlabfns. Accessed 25 August 2017.

  • Kurtulmus, F., Lee, W. S., & Vardar, A. (2014). Immature peach detection in colour images acquired in natural illumination conditions using statistical classifiers and neural network. Precision Agriculture,15(1), 57–79.

    Article  Google Scholar 

  • Li, H., Lee, W. S., & Wang, K. (2016). Immature green citrus fruit detection and counting based on fast normalized cross correlation (FNCC) using natural outdoor colour images. Precision Agriculture,17(6), 678–697.

    Article  Google Scholar 

  • Linker, R. (2017). A procedure for estimating the number of green mature apples in night-time orchard images using light distribution and its application to yield estimation. Precision Agriculture,18(1), 59–75.

    Article  Google Scholar 

  • Linker, R., Cohen, O., & Naor, A. (2012). Determination of the number of green apples in RGB images recorded in orchards. Computers and Electronics in Agriculture,81(1), 45–57.

    Article  Google Scholar 

  • Liu, H., Latecki, L. J., & Yan, S. (2010). Robust clustering as ensembles of affinity relations. Advances in Neural Information Processing Systems,23, 1414–1422.

    Google Scholar 

  • Low, K. (2004). Linear least-squares optimization for point-toplane ICP surface registration. Technical report, TR04-004, University of North Carolina, Chapel Hill, USA.

  • Lu, C., Latecki, L. J., Adluru, N., Yang, X., & Ling, H. (2009). Shape guided contour grouping with particle filters. In Proceedings of the IEEE 12th International Conference on Computer Vision (pp. 2288–2295). Los Alamitos, CA, USA: IEEE Computer Society Press.

  • Lu, J., Lee, W. S., Gan, H., & Hu, X. (2018). Immature citrus fruit detection based on local binary pattern feature and hierarchical contour analysis. Biosystems Engineering,171, 78–90.

    Article  Google Scholar 

  • Lu, J., & Sang, N. (2015). Detecting citrus fruits and occlusion recovery under natural illumination conditions. Computers and Electronics in Agriculture,110(C), 121–130.

    Article  Google Scholar 

  • Luo, L., Tang, Y., Zou, X., Wang, C., Zhang, P., & Feng, W. (2016). Robust grape cluster detection in a vineyard by combining the adaboost framework and multiple color components. Sensors,16(12), 2098.

    Article  Google Scholar 

  • Ma, T., & Latecki, L. J. (2011). From partial shape matching through local deformation to robust global shape similarity for object detection. In Proceedings of the 2011 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (pp. 1441–1448). Los Alamitos, CA, USA: IEEE Computer Society Press.

  • Maire, M., Arbelaez, P., Fowlkes, C., & Malik, J. (2008). Using contours to detect and localize junctions in natural images. Computer Vision and Pattern Recognition,11, 1–8.

    Google Scholar 

  • Maji, S., & Malik, J. (2009). Object detection using a max-margin Hough transform. In Proceedings of the 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (pp 1038–1045). Los Alamitos, CA, USA: IEEE Computer Society Press.

  • Murillo-Bracamontes, E. A., Martinez-Rosas, M. E., Miranda-Velasco, M. M., Martinez-Reyes, H. L., Martinez-Sandoval, J. R., & Cervantes-De-Avila, H. (2012). Implementation of Hough transform for fruit image segmentation. Procedia Engineering,35(12), 230–239.

    Article  Google Scholar 

  • Ommer, B., & Malik, J. (2009). Multi-scale object detection by clustering lines. In Proceedings of the IEEE 12th International Conference on Computer Vision (pp. 484–491). Los Alamitos, CA, USA: IEEE Computer Society Press.

  • Opelt, A., Pinz, A., & Zisserman, A. (2006). A boundary-fragment-model for object detection. In Proceedings of the 9th European Conference on Computer Vision (pp. 575–588). Heidelberg, Germany: Springer.

  • Payne, A., Walsh, K., Subedi, P., & Jarvis, D. (2014). Estimating mango crop yield using image analysis using fruit at ‘stone hardening’ stage and night time imaging. Computers and Electronics in Agriculture,100, 160–167.

    Article  Google Scholar 

  • Pearl, J. (1982). Reverend bayes on inference engines: a distributed hierarchical approach. In Proceedings of the Second AAAI Conference on Artificial Intelligence (pp. 133–136). Cambridge, UK: AAAI Press.

  • Puttemans, S., Vanbrabant, Y., Tits, L., & Goedeme, T. (2016). Automated visual fruit detection for harvest estimation and robotic harvesting. In Proceedings of the Sixth International Conference on Image Processing Theory, Tools and Applications (IPTA) (pp. 1–6). Los Alamitos, CA, USA: IEEE Computer Society Press.

  • Qureshi, W. S., Payne, A., Walsh, K. B., Linker, R., Cohen, O., & Dailey, M. N. (2016). Machine vision for counting fruit on mango tree canopies. Precision Agriculture,17(3), 1–21.

    Google Scholar 

  • Riemenschneider, H., Donoser, M., & Bischof, H. (2010). Using Partial Edge Contour Matches for Efficient Object Category Localization. In Proceedings of the 11th European Conference on Computer Vision (pp. 29–42). Heidelberg, Germany: Springer.

  • Shotton, J., Blake, A., & Cipolla, R. (2008). Multiscale categorical object recognition using contour fragments. IEEE Transactions on Pattern Analysis and Machine Intelligence,30(7), 1270–1281.

    Article  PubMed  Google Scholar 

  • Slaughter, D. C., & Harrell, R. C. (1989). Discriminating fruit for robotic harvest using color in natural outdoor scenes. Transactions of the ASAE,32(2), 0757–0763.

    Article  Google Scholar 

  • Su, Y., Liu, Y., Cuan, B., & Zheng, N. (2015). Contour guided hierarchical model for shape matching. In Proceedings of the 2015 IEEE International Conference on Computer Vision (pp. 1609–1617). Los Alamitos, CA, USA: IEEE Computer Society Press.

  • Tico, M., & Kuosmanen, P. (2003). Fingerprint matching using an orientation-based minutia descriptor. IEEE Transactions on Pattern Analysis and Machine Intelligence,25(8), 1009–1014.

    Article  Google Scholar 

  • Wachs, J. P., Stern, H. I., Burks, T., & Alchanatis, V. (2010). Low and high-level visual feature-based apple detection from multi-modal images. Precision Agriculture,11(6), 717–735.

    Article  Google Scholar 

  • Wang, Q., Nuske, S., Bergerman, M., & Singh, S. (2013). Automated crop yield estimation for apple orchards. In J. Desai, G. Dudek, O. Khatib, & V. Kumar (Eds.), Experimental robotics. Springer Tracts in Advanced Robotics (Vol. 88). Heidelberg, Germany: Springer.

    Google Scholar 

  • Wang, C., Zou, X., Tang, Y., Luo, L., & Feng, W. (2016). Localisation of litchi in an unstructured environment using binocular stereo vision. Biosystems Engineering,145, 39–51.

    Article  Google Scholar 

  • Xiang, R., Jiang, H., & Ying, Y. (2014). Recognition of clustered tomatoes based on binocular stereo vision. Computers and Electronics in Agriculture,106, 75–90.

    Article  Google Scholar 

  • Yu, Q., Wei, H., & Yang, C. (2017). Local part chamfer matching for shape-based object detection. Pattern Recognition,65, 82–96.

    Article  Google Scholar 

  • Zhao, C., Lee, W. S., & He, D. (2016). Immature green citrus detection based on colour feature and sum of absolute transformed difference (SATD) using colour images in the citrus grove. Computers and Electronics in Agriculture,124, 243–253.

    Article  Google Scholar 

  • Zou, X., Ye, M., Luo, C., Xiong, J., Luo, L., Wang, H., et al. (2016). Fault-tolerant design of a limited universal fruit-picking end-effector based on vision-positioning error. Applied Engineering in Agriculture,32(1), 5–18.

    Article  Google Scholar 

  • Zou, X., Zou, H., & Lu, J. (2012). Virtual manipulator-based binocular stereo vision positioning system and errors modelling. Machine Vision and Applications,23(1), 43–63.

    Article  Google Scholar 

Download references

Acknowledgements

This work was funded by a grant from the National Natural Science Foundation of China (No. 31571568), and the Project of Province Science and Technology of Guangdong (No. 2017A030222005).

Author information

Authors and Affiliations

  1. Key Laboratory of Key Technology on Agricultural Machine and Equipment, Ministry of Education, South China Agricultural University, 483 Wushan Road, Guangzhou, 510642, China

    Guichao Lin, Xiangjun Zou, Jiabing Cheng & Juntao Xiong

  2. College of Mechanical and Electrical Engineering, Chuzhou University, 1 Huifeng West Road, Chuzhou, 239000, China

    Guichao Lin

  3. College of Urban and Rural Construction, Zhongkai University of Agriculture and Engineering, Guang Xin Road, Guangzhou, 510225, China

    Yunchao Tang

Authors
  1. Guichao Lin
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yunchao Tang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xiangjun Zou
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jiabing Cheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Juntao Xiong
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Yunchao Tang or Xiangjun Zou.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lin, G., Tang, Y., Zou, X. et al. Fruit detection in natural environment using partial shape matching and probabilistic Hough transform. Precision Agric 21, 160–177 (2020). https://doi.org/10.1007/s11119-019-09662-w

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11119-019-09662-w

Keywords

Search

Navigation