Skip to main content
SpringerLink
Log in

Semantic Image Segmentation Method with Multiple Adjacency Trees and Multiscale Features

  • Published:
Cognitive Computation Aims and scope Submit manuscript

Abstract

Semantic image segmentation is the basis of image understanding, which is one of the most important human cognitive activities. Cognitive studies have shown that human neocortical information transmission depends on cognitive processing at multiple scales, and contextual information aids the human cognitive system in solving perceptual inference tasks. Inspired by multiscale cognitive mechanisms and contextual effects, in this paper, we propose a semantic image segmentation method addressing multiscale features and contextual information. To integrate multiscale features, after over-segmenting an image into small-scale segments, we employ a segment-based classifier and a CRF (conditional random field) model to generate large-scale regions. We then use the features of regions to train a region-based classifier. To capture context, we propose a multiple adjacency tree model where each tree represents one type of region relevance and can be generated by the adjacency graph corresponding to that relevance metric. Using the multiple tree model instead of a general graph model, we can perform exact inference with some simple assumptions and capture multiple types of regional context dependency. Experiments on the MSRC-21 and Stanford background datasets show advantages of our method over a segment-based CRF model using single-scale features. The results demonstrate the importance of multiscale features and contextual information.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

References

  1. Chang CC, Lin CJ. 2011. LIBSVM: a library for support vector machines, Vol. 2. Software available at http://www.csie.ntu.edu.tw/∼cjlin/libsvm.

  2. Criminisi A. 2005. The whole MLP team: Microsoft research Cambridge msrc-v2 image database(21 object classes). http://research.microsoft.com/vision/cambridge/recognition/.

  3. Fulkerson B, Vedaldi A, Soatto S. Class segmentation and object localization with superpixel neighborhoods. International conference on computer vision (ICCV). IEEE Computer Society; 2009. p. 670–677.

  4. Galleguillos C, Belongie S. Context based object categorization: a critical survey. Comput Vis Image Underst 2010;114(6):712–722.

  5. Haikonen POA. The role of associative processing in cognitive computing. Cogn Comput 2009;1:42–49.

    Article  Google Scholar 

  6. Han D, Hu Y, Ai S, Wang G. Uncertain graph classification based on extreme learning machine. Cogn Comput 2015;7:346–358.

    Article  Google Scholar 

  7. He X. 2008. Learning structured prediction models for image labeling. Ph.D. thesis, University of Toronto.

  8. Huang GB. An insight into extreme learning machines: random neurons, random features and kernels. Cogn Comput 2014;6:376–390.

    Article  Google Scholar 

  9. Huang GB, Zhou H, Ding X, Zhang R. Extreme learning machine for regression and multiclass classification. IEEE Trans Pattern Anal Mach Intell 2012;42(2):513–529.

    Google Scholar 

  10. Huang Q, Han M, Wu B. Ioffe, S.: A hierarchical conditional random field for labeling and segmenting images of street scenes. IEEE conference on computer vision and pattern recognition (CVPR); 2011. p. 1953–1960.

  11. Ingber L. Computational algorithms derived from multiple scales of neocortical processing. Cogn Comput 2012; 4:38–50.

    Article  Google Scholar 

  12. Kae A, Sohn K, Lee H, Learned-Miller E. Augmenting CRFs with Boltzmann machine shape priors for image labeling. IEEE conference on computer vision and pattern recognition (CVPR); 2013. p. 2019–2026.

  13. Kohli P, Ladicky L, Torr PHS. Robust higher order potentials for enforcing label consistency. Int J Comput Vis 2009;82(3):302–324.

    Article  Google Scholar 

  14. Ladicky L, Russell C, Kohli P. Associative hierarchical CRFs for object class image segmentation. International conference on computer vision (ICCV). IEEE Computer Society; 2009. p. 739–746.

  15. Ladicky L, Russell C, Kohli P, Torr PHS. Graph cut based inference with co-occurrence statistics. European conference on computer vision (ECCV); 2010. p. 239–253.

  16. Liu F, Lin G, Shen C. CRF learning with CNN features for image segmentation. Pattern Recogn 2015; 48:2983–2992.

    Article  Google Scholar 

  17. Liu W, Tao D, Cheng J, Tang Y. Multiview hessian discriminative sparse coding for image annotation. Comput Vis Image Underst 2014;118:50–60.

    Article  Google Scholar 

  18. Luo Y, Tao D, Xu C. Multiview matrix completion for multilabel image classification. IEEE Trans Image Process 2015;24(8):2355–2367.

    Article  Google Scholar 

  19. Mottaghi R, Chen X, Liu X, Cho NG, Lee SW. The role of context for object detection and semantic segmentation in the wild. IEEE conference on computer vision and pattern recognition (CVPR); 2014.

  20. Mottaghi R, Fidler S, Yao J, Urtasun R, Parikh D. Analyzing semantic segmentation using hybrid human-machine CRFs. IEEE conference on computer vision and pattern recognition (CVPR); 2013. p. 3143–3150.

  21. Nematollahi M, Zhang XP. A new robust context-based dense crf model for image labeling. International conference on image processing (ICIP); 2014. p. 5876–5880.

  22. Nowozin S, Gehler PV, Lampert CH. On parameter learning in crf-based approaches to object class image segmentation. European conference on computer vision (ECCV); 2010. p. 98–111.

  23. Ogiela L, Ogiela MR. Cognitive approach to bio-inspired medical image understanding. IEEE fifth conference on bio-inspired computing: theories and applications; 2010. p. 1010–1013.

  24. Parikh D, Zitnick CL, Chen T. Exploring tiny images: the roles of appearance and contextual information for machine and human object recognition. IEEE Trans Pattern Anal Mach Intell 2014;34(10):1978–1991.

  25. Pieck MA, van der Sommen F, Zinger S, de With PH. Real-time semantic context labeling for image understanding. International conference on image processing (ICIP). IEEE Computer Society; 2015. p. 3180–3184.

  26. Sato YD, Nagatomi T, Horio K, Miyamoto H. The cognitive mechanisms of multi-scale perception for the recognition of extremely similar faces. Cogn Comput 2015;7:501–508.

    Article  Google Scholar 

  27. Gould S., Fulton R., Koller D. Decomposing a scene into geometric and semantically consistent regions. IEEE international conference on computer vision (ICCV). IEEE Computer Society; 2009.

  28. Shotton J, Winn J, Rother C, Criminisi A. TextonBoost for image understanding: multi-class object recognition and segmentation by jointly modeling texture, layout, and context. Int J Comput Vis 2007;81(1):2–23.

    Article  Google Scholar 

  29. Szummer M, Kohli P, Hoiem D. Learning CRFs using graph cuts. European conference on computer vision (ECCV); 2008. p. 582–595.

  30. Tang K, Paluri M, Fei-Fei L, Fergus R, Bourdev L. Improving image classification with local context. International conference on computer vision (ICCV). IEEE Computer Society; 2015. p. 1008–1016.

  31. Vedaldi A, Fulkerson B. 2008. Vlfeat: an open and portable library of computer vision algorithms. http://www.vlfeat.org/.

  32. Vedaldi A, Soatto S. Quick shift and kernel methods for mode seeking. European conference on computer vision (ECCV); 2008.

  33. Wang X, Song Y, Zhang Y, Xin J. Natural scene text detection with multi-layer segmentation and higher order conditional random field based analysis. Pattern Recogn Lett 2015;60:41– 47.

    Article  Google Scholar 

  34. Xu C, Tao D, Xu C. Large-margin multi-view information bottleneck. IEEE Trans Pattern Anal Mach Intell 2014;36(8):1559–1572.

    Article  PubMed  Google Scholar 

  35. Xu L, Ding S, Xu X, Zhang N. Self-adaptive extreme learning machine optimized by rough set theory and affinity propagation clustering. Cogn Comput 2016;4:1–9.

    Google Scholar 

  36. Yu L, Xie J, Chen S. Conditional random field-based image labeling combining features of pixels, segments and regions. IET Comput Vis 2012;6(5):459–467.

    Article  Google Scholar 

  37. Zhang P, Li M, Wu Y, An J, Jia J. Unsupervised SAR images segmentation using high-order conditional random fields model based on product-of-experts. Pattern Recogn Lett 2016;78:48–55.

    Article  Google Scholar 

  38. Zhang P, Li M, Wu Y., Li H. Hierarchical conditional random fields model for semisupervised SAR image segmentation. IEEE Trans Geosci Remote Sens 2015;53(9):4933–4941.

    Article  Google Scholar 

  39. Zhao J, Du C, Sun H, Liu X, Sun J. Biologically motivated model for outdoor scene classification. Cogn Comput 2015;7:20–33.

    Article  Google Scholar 

  40. Zhu H, Meng F, Cai J, Lu S. Beyond pixels: A comprehension survey from bottom-up to semantic image segmentation and cosegmentation. J Vis Commun Image Represent 2016;34:12–27.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. College of Command Information System, PLA University of Science and Technology, Nanjing, 210007, China

    Jun Xie

  2. Institute of Communications Engineering, PLA University of Science and Technology, Nanjing, 210007, China

    Lu Yu & Lei Zhu

  3. College of Science, Nanjing University of Aeronautics and Astronautics, Nanjing, 210016, China

    Xiaohong Chen

Authors
  1. Jun Xie
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Lu Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Lei Zhu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xiaohong Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Lu Yu.

Ethics declarations

This work is supported by the National Natural Science Foundation of China 61101202, 61403193, 61375057 and the Natural Science Foundation of Jiangsu Province BK20140065.

Conflict of Interests

All authors declare that they have no conflict of interest.

Ethical Approval

This article does not contain any studies with human participants or animals performed by any of the authors.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Xie, J., Yu, L., Zhu, L. et al. Semantic Image Segmentation Method with Multiple Adjacency Trees and Multiscale Features. Cogn Comput 9, 168–179 (2017). https://doi.org/10.1007/s12559-016-9441-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12559-016-9441-5

Keywords

Search

Navigation