Skip to main content
SpringerLink
Log in

Building and using fuzzy multimedia ontologies for semantic image annotation

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

This paper proposes a methodology for building fuzzy multimedia ontologies dedicated to image annotation. The built ontology incorporates visual, conceptual, contextual and spatial knowledge about image concepts in order to model image semantics in an effective way. Indeed, our approach uses visual and conceptual information to build a semantic hierarchy that will serve as a backbone of our ontology. Contextual and spatial information about image concepts are then computed and incorporated in the ontology in order to model richer semantic relationships between these concepts. Fuzzy description logics are used as a formalism to represent our ontology and the inherent uncertainty and imprecision of this kind of information. Subsequently, we propose a new approach for image annotation based on hierarchical image classification and a multi-stage reasoning framework for reasoning about the consistency of the produced annotation. In this approach, fuzzy ontological reasoning is used in order to achieve a semantically relevant decision on the belonging of a given image to the set of concepts from the annotation vocabulary. An empirical evaluation of our approach on Pascal VOC’2009 and Pascal VOC’2010 datasets has shown a significant improvement on the average precision results.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

Notes

  1. http://owlapi.sourceforge.net/index.html

  2. n, m are natural numbers, such that n ≥ 0, m > 0. d is an unary fuzzy domain predicate.

  3. A candidate annotation \(\mathcal{P}\) consists of a set of candidate concepts {\(c_j \in \mathcal C \cup \mathcal C', j=1..n_{i_i}\)} and their confidence values {\(\alpha_j, j=1..n_{i_i}\)}, predicted as describing the image content.

References

  1. Baader F, Calvanese D, McGuinness DL, Nardi D, Patel-Schneider PF (eds) (2003) The description logic handbook: theory, implementation, and applications

  2. Bannour H, Hudelot C (2011) Towards ontologies for image interpretation and annotation. In: Content-based multimedia indexing (CBMI’11)

  3. Bannour H, Hudelot C (2012) Building semantic hierarchies faithful to image semantics. In: International conference on advances in multimedia modeling (MMM’12), pp 4–15

  4. Bannour H, Hudelot C (2012) Hierarchical image annotation using semantic hierarchies. In: Proceedings of the 21st ACM international conference on information and knowledge management (CIKM’12), pp 2431–2434

  5. Barnard K, Duygulu P, Forsyth D, de Freitas N, Blei DM, Jordan MI (2003) Matching words and pictures. J Mach Learn Res 3:1107–1135

    MATH  Google Scholar 

  6. Bart E, Porteous I, Perona P, Welling M (2008) Unsupervised learning of visual taxonomies. In: Computer vision and pattern recognition (CVPR)

  7. Bloch I (2005) Fuzzy spatial relationships for image processing and interpretation: a review. Image Vis Comput 23(2):89–110

    Article  Google Scholar 

  8. Bobillo F, Straccia U (2011) Reasoning with the finitely many-valued lukasiewicz fuzzy description logic sroiq. Inform Sci 181(4):758–778

    Article  MATH  MathSciNet  Google Scholar 

  9. Carneiro G, Chan AB, Moreno PJ, Vasconcelos N (2007) Supervised learning of semantic classes for image annotation and retrieval. IEEE Trans Pattern Anal Mach Intell 29(3):394–410

    Article  Google Scholar 

  10. Choi MJ, Lim J, Torralba A, Willsky A (2010) Exploiting hierarchical context on a large database of object categories. In: Computer vision and pattern recognition (CVPR), pp 129–136

  11. Cortes C, Vapnik V (1995) Support-vector networks. Mach Learn 20(3):273–297

    MATH  Google Scholar 

  12. Dasiopoulou S, Kompatsiaris I, Strintzis M (2009) Applying fuzzy DLs in the extraction of image semantics. In: Spaccapietra S, Delcambre L (eds) Journal on data semantics XIV. Lecture notes in computer science, vol 5880. Springer Berlin, Heidelberg, pp 105–132

    Chapter  Google Scholar 

  13. Dasiopoulou S, Tzouvaras V, Kompatsiaris I, Strintzis MG (2010) Enquiring mpeg-7 based multimedia ontologies. Multimed Tools Appl 46:331–370

    Article  Google Scholar 

  14. Deng J, Dong W, Socher R, Li LJ, Li K, Fei-Fei L (2009) Imagenet: a large-scale hierarchical image database. In: Computer vision and pattern recognition (CVPR)

  15. Everingham M, Van Gool L, Williams CKI, Winn J, Zisserman A (2009) The PASCAL visual object classes challenge 2009 (VOC2009) results. http://www.pascal-network.org/challenges/VOC/voc2009/workshop/index.html

  16. Everingham M, Van Gool L, Williams CKI, Winn J, Zisserman A (2010) The PASCAL visual object classes challenge 2010 (VOC2010) results. http://www.pascal-network.org/challenges/VOC/voc2010/workshop/index.html

  17. Fan J, Gao Y, Luo H (2008) Integrating concept ontology and multitask learning to achieve more effective classifier training for multilevel image annotation. IEEE Trans Image Process 17(3):407–426

    Article  MathSciNet  Google Scholar 

  18. Griffin G, Perona P (2008) Learning and using taxonomies for fast visual categorization. In: Computer vision and pattern recognition (CVPR)

  19. Gruber TR (1995) Toward principles for the design of ontologies used for knowledge sharing. Int J Hum-Comput Stud 43(5):907–928

    Article  Google Scholar 

  20. Gupta A, Mannem P (2012) From image annotation to image description. Neural Inf Process 7667:196–204

    Article  Google Scholar 

  21. Hauptmann A, Yan R, Lin WH (2007) How many high-level concepts will fill the semantic gap in news video retrieval? In: International conference on image and video retrieval (CIVR)

  22. Hollink L, Nguyen G, Schreiber G, Wielemaker J, Wielinga B, Worring M (2004) Adding spatial semantics to image annotations. In: International workshop on knowledge markup and semantic annotation

  23. Horridge M, Bechhofer S (2011) The owl api: a java api for owl ontologies. Semant Web 2(1):11–21

    Google Scholar 

  24. Hudelot C, Atif J, Bloch I (2008) Fuzzy spatial relation ontology for image interpretation. Fuzzy Set Syst 159:1929–1951

    Article  MathSciNet  Google Scholar 

  25. Hudelot C, Atif J, Bloch I (2010) Integrating bipolar fuzzy mathematical morphology in description logics for spatial reasoning. In: European conference on artificial intelligence (ECAI), pp 497–502

  26. Kompatsiaris Y, Hobson P (2008) Semantic multimedia and ontologies: theory and applications. Springer

  27. Lavrenko V, Manmatha R, Jeon J (2003) A model for learning the semantics of pictures. In: Neural information processing systems. MIT, Cambridge

    Google Scholar 

  28. Li FF, Perona P (2005) A bayesian hierarchical model for learning natural scene categories. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR’05), vol 2. Washington, DC, USA, pp 524–531

  29. Li LJ, Wang C, Lim Y, Blei DM, Li FF (2010) Building and using a semantivisual image hierarchy. In: Computer vision and pattern recognition (CVPR)

  30. Liu Y, Zhang D, Lu G, Ma WY (2007) A survey of content-based image retrieval with high-level semantics. Pattern Recogn 40(1):262–282

    Article  MATH  Google Scholar 

  31. Lowe DG (1999) Object recognition from local scale-invariant features. In: International conference on computer vision (ICCV)

  32. Marszalek M, Schmid C (2007) Semantic hierarchies for visual object recognition. In: Computer vision and pattern recognition (CVPR)

  33. Simou N, Tzouvaras V, Avrithis Y, Stamou G, Kollias S (2005) A visual descriptor ontology for multimedia reasoning. In: WIAMIS

  34. Simou N, Athanasiadis T, Stoilos G, Kollias SD (2008) Image indexing and retrieval using expressive fuzzy description logics. Signal Image Video Process 2(4):321–335

    Article  Google Scholar 

  35. Smeulders AWM, Worring M, Santini S, Gupta A, Jain R (2000) Content-based image retrieval at the end of the early years. IEEE Trans Pattern Anal Mach Intell 22(12):1349–1380

    Article  Google Scholar 

  36. Spaccapietra S, Cullot N, Parent C, Vangenot C (2004) On spatial ontologies. In: Brazilian symposium on geoinformatics

  37. Stoilos G, Stamou GB (2007) Extending fuzzy description logics for the semantic web. In: Workshop on OWL: experiences and directions (OWLED)

  38. Straccia U (2001) Reasoning within fuzzy description logics. J Artif Intell Res 14:137–166

    MATH  MathSciNet  Google Scholar 

  39. Straccia U (2006) A fuzzy description logic for the semantic web. In: Sanchez E (ed) Fuzzy logic and the semantic web. Capturing intelligence, vol 1. Elsevier, pp 73–90

  40. Straccia U (2010) An ontology mediated multimedia information retrieval system. In: Multiple-valued logic (ISMVL), pp 319–324

  41. Straccia U (2012) Description logics with fuzzy concrete domains. In: Computing research repository (CoRR). arXiv:abs/1207.1410

  42. Tousch AM, Herbin S, Audibert JY (2012) Semantic hierarchies for image annotation: a survey. Pattern Recogn 45(1):333–345

    Article  Google Scholar 

  43. Wu L, Hua XS, Yu N, Ma WY, Li S (2012) Flickr distance: a relationship measure for visual concepts. IEEE Trans Pattern Anal Mach Intell 34(5):863 –875

    Article  Google Scholar 

  44. Xiao J, Hays J, Ehinger KA, Oliva A, Torralba A (2010) Sun database: large-scale scene recognition from abbey to zoo. In: Computer vision and pattern recognition (CVPR). IEEE, pp 3485–3492

  45. Yang J, Yu K, Huang T (2010) Efficient highly over-complete sparse coding using a mixture model. In: Proceedings of the 11th European conference on computer vision: part V, ECCV’10, pp 113–126

  46. Yao B, Yang X, Lin L, Lee MW, Zhu SC (2010) I2t: Image parsing to text description. Proc IEEE 98(8):1485–1508

    Article  Google Scholar 

  47. Zhou X, Yu K, Zhang T, Huang T (2010) Image classification using super-vector coding of local image descriptors. In: European conference on computer vision (ECCV)

Download references

Author information

Authors and Affiliations

  1. MAS Laboratory, Ecole Centrale Paris, 92 295, Chatenay-Malabry, France

    Hichem Bannour & Céline Hudelot

Authors
  1. Hichem Bannour
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Céline Hudelot
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hichem Bannour.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Bannour, H., Hudelot, C. Building and using fuzzy multimedia ontologies for semantic image annotation. Multimed Tools Appl 72, 2107–2141 (2014). https://doi.org/10.1007/s11042-013-1491-z

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-013-1491-z

Keywords

Search

Navigation