Skip to main content
Log in

Enabling context-aware multimedia annotation by a novel generic semantic problem-solving platform

Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Automatic generation of metadata, facilitating the retrieval of multimedia items, potentially saves large amounts of manual work. However, the high specialization degree of feature extraction algorithms makes them unaware of the context they operate in, which contains valuable and often necessary information. In this paper, we show how Semantic Web technologies can provide a context that algorithms can interact with. We propose a generic problem-solving platform that uses Web services and various knowledge sources to find solutions to complex requests. The platform employs a reasoner-based composition algorithm, generating an execution plan that combines several algorithms as services. It then supervises the execution of this plan, intervening in case of errors or unexpected behavior. We illustrate our approach by a use case in which we annotate the names of people depicted in a photograph.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Listing 1
Fig. 1
Listing 2
Listing 3
Listing 4
Listing 5
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

References

  1. Atrey PK, Hossain MA, El Saddik A, Kankanhalli MS (2010) Multimodal fusion for multimedia analysis: a survey. Multimedia Syst 16(6):345–379. doi:10.1007/s00530-010-0182-0

    Article  Google Scholar 

  2. Berners-Lee T, Connolly D (2009) Notation3 (N3): a readable RDF syntax. W3C recommendation. http://www.w3.org/TeamSubmission/n3/. Accessed 12 December 2010

  3. Berners-Lee T, Connolly D, Kagal L, Scharf Y, Hendler J (2008) N3Logic: a logical framework for the World Wide Web. Theory and Practice of Logic Programming 8(3):249–269. doi:10.1017/S1471068407003213

    Article  MathSciNet  MATH  Google Scholar 

  4. Berners-Lee T, Hendler J, Lassila O (2001) The Semantic Web. Sci Am 284(5):34

    Article  Google Scholar 

  5. Bizer C, Heath T, Berners-Lee T (2009) Linked Data - the story so far. Int J Semant Web Inf 5(3):1–22. doi10.4018/jswis.2009070101

    Article  Google Scholar 

  6. Bloehdorn S, Petridis K, Saathoff C, Simou N, Tzouvaras V, Avrithis Y, Handschuh S, Kompatsiaris Y, Staab S, Strintzis M (2005) Semantic annotation of images and videos for multimedia analysis. In: The Semantic Web: research and applications, pp 592–607

  7. Corkill DD (1991) Blackboard systems. AI Expert 6(9):40–47

    Google Scholar 

  8. DBpedia (2010) http://dbpedia.org/. Accessed 12 December 2010

  9. De Roo J (2010) Euler proof mechanism. http://eulersharp.sourceforge.net/. Accessed 12 December 2010

  10. Fensel D, Bussler C (2002) Semantic Web enabled Web services. In: 2nd annual diffuse conference

  11. Finzi A, Pirri F, Reiter R (2000) Open world planning in the situation calculus. In: Proceedings of the national conference on artificial intelligence, pp 754–760

  12. Freebase (2010) http://www.freebase.com/. Accessed 12 December 2010

  13. Hanjalic A, Lienhart R, Ma WY, Smith RJ (2008) The holy grail of multimedia information retrieval: so close or yet so far away? In: Proceedings of the IEEE Multimedia, vol 96(4), pp 541–547. http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=4457924

  14. Horrocks I, Patel-Schneider PF, Boley H, Tabet S (2004) SWRL: a Semantic Web rule language combining OWL and RuleML. W3C member submission. http://www.w3.org/Submission/SWRL/. Accessed 12 December 2010

  15. Hristoskova A, Volckaert B, Turck FD (2009) Dynamic composition of semantically annotated web services through QoS-aware HTN planning algorithms. In: ICIW ’09: Proceedings of the 2009 fourth international conference on Internet and Web applications and services. IEEE Computer Society, Washington, pp 377–382

  16. Junghans M, Agarwal S, Studer R (2010) Towards practical Semantic Web service discovery. In: The Semantic Web: research and applications, 7th extended Semantic Web conference, ESWC 2010, Heraklion, Crete, Greece, May 30–June 3, 2010, Proceedings. Springer, New York

  17. Klusch M, Fries B, Khalid M (2005) OWLS-MX: Hybrid OWL-S service matchmaking. In: Proceedings of 1st intl. AAAI fall symposium on agents and the Semantic Web

  18. Klyne G, Carrol JJ (2004) Resource description framework (RDF): concepts and abstract syntax. W3C recommendation. http://www.w3.org/TR/2004/REC-rdf-concepts-20040210/. Accessed 12 December 2010

  19. Li L, Horrocks I (2003) A software framework for matchmaking based on Semantic Web technology. In: WWW ’03: Proceedings of the 12th international conference on World Wide Web. ACM Press, New York, pp 331–339. doi:10.1145/775152.775199

    Google Scholar 

  20. Martin D, Burstein M, Hobbs J, Lassila O (2004) OWL-S: Semantic markup for web services. W3C member submission. http://www.w3.org/Submission/OWL-S/. Accessed 12 December 2010

  21. Medjahed B, Bouguettaya A, Elmagarmid AK (2003) Composing web services on the Semantic Web. VLDB J 12(4):333–351. doi:10.1007/s00778-003-0101-5

    Article  Google Scholar 

  22. Osmun T (2010) Euler Eye installation, demo, and deep taxonomy benchmark. http://ruleml.org/WellnessRules/files/WellnessRulesN3-2009-11-10.pdf. Accessed 12 December 2010

  23. Osmun T (2010) Eye deep taxonomy benchmark results. http://eulersharp.sourceforge.net/2003/03swap/dtb-2010.txt. Accessed 12 December 2010

  24. Prud’hommeaux E, Seaborne A (2008) SPARQL query language for RDF. W3C recommendation. http://www.w3.org/TR/rdf-sparql-query/. Accessed 12 December 2010

  25. Rahurkar M, Tsai S, Dagli C, Huang T (2010) Image interpretation using large corpus: wikipedia. Proc IEEE 98(8):1509–1525. doi:10.1109/JPROC.2010.2050410. http://ieeexplore.ieee.org/xpl/freeabs_all.jsp?arnumber=5484723

    Article  Google Scholar 

  26. Redavid D, Iannone L, Payne T (2007) OWL-S atomic services composition with SWRL rules. In: Proceedings of the 4th Italian Semantic Web workshop. http://eprints.ecs.soton.ac.uk/15658/

  27. Roy PV, Haridi S (2004) Concepts, techniques, and models of computer programming. MIT Press, Cambridge

    Google Scholar 

  28. Shin D, Lee K, Suda T (2009) Automated generation of composite Web services based on functional semantics. Web Semantics: Science

    Google Scholar 

  29. Smith JR, Schirling P (2006) Metadata standards roundup. IEEE MultiMedia 13:84–88. doi:10.1109/MMUL.2006.34

    Article  Google Scholar 

  30. Sycara K, Paolucci M, Ankolekar A, Srinivasan N (2003) Automated discovery, interaction and composition of Semantic Web services. Journal of Web Semantics 1(1):27–46

    Article  Google Scholar 

  31. Troncy R, Mannens E, Pfeiffer S, Van Deursen D (2010) Media fragments URI 1.0. W3C working draft. http://www.w3.org/TR/media-frags/. Accessed 12 December 2010

  32. Verborgh R (2010) Generic semantic problem-solving platform demonstration: road trip. http://multimedialab.elis.ugent.be/organon/arseco/demonstrations. Accessed 12 December 2010

  33. Verborgh R, Van Deursen D, De Roo J, Mannens E, Van de Walle R (2010) SPARQL endpoints as front-end for multimedia processing algorithms. In: Proceedings of the fourth international workshop on service matchmaking and resource retrieval in the Semantic Web at the 9th international Semantic Web conference (ISWC 2010)

  34. Verstockt S, Van Leuven S, Van de Walle R, Dermaut E, Torelle S, Gevaert W (2009) Actor recognition for interactive querying and automatic annotation in digital video. In: 13th, Proceedings of the IASTED international conference on Internet and multimedia systems and applications. ACTA Press, Honolulu, pp 149–155

    Google Scholar 

  35. Viola P, Jones MJ (2004) Robust real-time face detection. Int J Comput Vis 57(2):137–154. doi:10.1023/B:VISI.0000013087.49260.fb

    Article  Google Scholar 

Download references

Acknowledgements

The research activities as described in this paper were funded by Ghent University, the Interdisciplinary Institute for Broadband Technology (IBBT), the Institute for the Promotion of Innovation by Science and Technology in Flanders (IWT), the Fund for Scientific Research Flanders (FWO Flanders), and the European Union.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Ruben Verborgh.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Verborgh, R., Van Deursen, D., Mannens, E. et al. Enabling context-aware multimedia annotation by a novel generic semantic problem-solving platform. Multimed Tools Appl 61, 105–129 (2012). https://doi.org/10.1007/s11042-010-0709-6

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-010-0709-6

Keywords

Navigation