Skip to main content
SpringerLink
Log in

Movie summarization using bullet screen comments

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Automatic movie summarization helps users to skim a movie in an efficient way. However, it is challenging because it requires the computer to automatically understand the movie content and the users’ opinions. Most previous works rely on the movie data itself without considering the opinions of the audience. In this paper, a novel approach for automatic movie summarization is presented by exploring a new type of user-generated data, i.e. bullet screen comments, which allow the audience to comment on the movie in a real-time manner. The number of the comments on a movie segment shows the exciting degree of the audience, while the content of the comments includes the concepts (e.g. the characters and the scenes) that interest the audience. In our approach, given a movie, bullet screen comments are utilized to select candidate highlight segments which are the most commented. Then the candidates are scored based on the number and the content of the bullet screen comments. Visual diversity is also considered in the scoring process. Finally, a subset of candidates which achieves the highest score is selected to compose a summary. Our experiments carried out on movies of different genres have shown the effectiveness of our proposed approach.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

References

  1. Agnihotri L, Devera KV, McGee T, Dimitrova N (2001) Summarization of video programs based on closed captions. Storage Retriev Media Databases. doi:10.1117/12.410973

  2. Bilibili. http://www.bilibili.com/

  3. Bilibilijj. http://www.bilibilijj.com/

  4. Darabi K, Ghinea G (2016) User-centred personalised video abstraction approach adopting SIFT features. Multimed Tools Appl. doi:10.1007/s11042-015-3210-4

  5. Dimoulas CA, Symeonidis AL (2015) Syncing shared multimedia through audiovisual bimodal segmentation. IEEE MultiMed 22(3):26–42. doi:10.1109/MMUL.2015.33

    Article  Google Scholar 

  6. Dogra DP, Ahmed A, Bhaskar H (2015) Smart video summarization using mealy machine-based trajectory modelling for surveillance applications. Multimed Tools Appl 75(11):6373–6401. doi:10.1007/s11042-015-2576-7

    Article  Google Scholar 

  7. Doman K, Tomita T, Ide I, Deguchi D, Murase H (2014) Event detection based on twitter enthusiasm degree for generating a sports highlight video. In: Proceedings of the 18th ACM international conference on multimedia. doi:10.1145/2647868.2654973

  8. Ejaz N, Mehmood I, Baik SW (2013) Efficient visual attention based framework for extracting key frames from videos. Signal Process Image Commun 28(1):34–44. doi:10.1016/j.image.2012.10.002

    Article  Google Scholar 

  9. Ejaz N, Mehmood I, Baik SW (2014) Feature aggregation based visual attention model for video summarization. Comput Electric Eng 40(3):993–1005. doi:10.1016/j.compeleceng.2013.10.005

    Article  Google Scholar 

  10. Evangelopoulos G, Zlatintsi A, Potamianos A, Maragos P, Rapantzikos K, Skoumas G, Avrithis Y (2013) Multimodal saliency and fusion for movie summarization based on aural, visual, and textual attention. IEEE Trans Multimed 15(7):1553–1568. doi:10.1109/TMM.2013.2267205

    Article  Google Scholar 

  11. Ferreira L, Silva Cruz LA, Assuncao P (2015) A generic framework for optimal 2D/3D key-frame extraction driven by aggregated saliency maps. Signal Process Image Commun 39:98–110. doi:10.1016/j.image.2015.09.005

    Article  Google Scholar 

  12. Furini M, Ghini V (2006) An audio-video summarization scheme based on audio and video analysis. In: Proceedings of IEEE consumer communications & networking. doi:10.1109/CCNC.2006.1593230

  13. Hannon J, McCarthy K, Lynch J, Smyth B (2011) Personalized and automatic social summarization of events in video. In: Proceedings of the 16th international conference on intelligent user interfaces. doi:10.1145/1943403.1943459

  14. Li Y, Lee SH, Yeh CH, Kuo CC (2006) Techniques for movie content analysis and skimming: tutorial and overview on video abstraction techniques. IEEE Signal Process Mag 23(2):79–89. doi:10.1109/MSP.2006.1621451

    Article  Google Scholar 

  15. Lin W, Sun MT, Li H, Chen Z, Li W, Zhou B (2012) Macroblock classification for video applications involving motions. IEEE Trans Broadcast 58 (1):34–46. doi:10.1109/TBC.2011.2170611

    Article  Google Scholar 

  16. Lin W, Zhang Y, Lu J, Zhou B, Wang J, Zhou Y (2015) Summarizing surveillance videos with local-patch-learning-based abnormality detection, blob sequence optimization, and type-based synopsis. Neurocomputing 155:84–98. doi:10.1016/j.neucom.2014.12.044

    Article  Google Scholar 

  17. Lowe DG (1999) Object recognition from local scale-invariant features. In: Proceedings of the 7th IEEE international conference on computer vision. doi:10.1109/ICCV.1999.790410

  18. Lu Z, Grauman K (2013) Story-driven summarization for egocentric video. In: Proceedings of IEEE conference on computer vision and pattern recognition. doi:10.1109/CVPR.2013.350

  19. Ma YF, Hua XS, Lu L, Zhang HJ (2005) A generic framework of user attention model and its application in video summarization. IEEE Trans Multimed 7(5):907–919. doi:10.1109/TMM.2005.854410

    Article  Google Scholar 

  20. Mei S, Guan G, Wang Z, He M, Hua X, Feng D (2014) L 2,0 constrained sparse dictionary selection for video summarization. In: International conference on multimedia & Expo. doi:10.1109/ICME.2014.6890179

  21. Mei S, Guan G, Wang Z, Wan S, He M, Feng D (2015) Video summarization via minimum sparse reconstruction. Pattern Recogn 48(2):522–533. doi:10.1016/j.patcog.2014.08.002

    Article  Google Scholar 

  22. Niconico. http://www.nicovideo.jp/

  23. Over P, Smeaton AF, Kelly P (2007) The TRECVID 2007 BBC rushes summarization evaluation pilot. In: Proceedings of the TRECVID workshop on video summarization. doi:10.1145/1290031.1290032

  24. Panagiotakis C, Doulamis A, Tziritas G (2009) Equivalent key frames selection based on iso-content principles. IEEE Trans Circ Syst Vid Technol 19(3):447–451. doi:10.1109/TCSVT.2009.2013517

    Article  Google Scholar 

  25. Peng J, Lin Q (2007) Keyframe-based video summary using visual attention clues. IEEE Multimed 17(2):64–73. doi:10.1109/MMUL.2009.65

    Google Scholar 

  26. Pickering MJ, Wong L, Rger SM (2003) ANSES: summarization of news video. Image Vid Retriev. doi:10.1007/3-540-45113-7_42

  27. Pritch Y, Rav-Acha A, Peleg S (2008) Nonchronological video synopsis and indexing. IEEE Trans Pattern Anal Mach Intell 30(11):1971–1984. doi:10.1109/TPAMI.2008.29

    Article  Google Scholar 

  28. Rapantzikos K, Evangelopoulos G, Maragos P, Avrithis YS (2007) An audio-visual saliency model for movie summarization. In: IEEE 9th Workshop on multimedia signal processing. doi:10.1109/MMSP.2007.4412882

  29. Sang J, Xu C (2010) Character-based movie summarization. In: Proceedings of the 18th ACM international conference on multimedia. doi:10.1145/1873951.1874096

  30. Shamma D, Kennedy L, Churchill E (2010) Summarizing media through short-messaging services. In: ACM Conference on computer supported cooperative work.

  31. Song Y, Vallmitjana J, Stent A, Jaimes A (2015) TVSum: summarizing web videos using titles. In: Proceedings of the IEEE conference on computer vision and pttern recognition. doi:10.1109/CVPR.2015.7299154

  32. Taskiran CM, Pizlo Z, Amir A, Ponceleon D, Delp EJ (2006) Automated video program summarization using speech transcripts. IEEE Trans Multimed 8 (4):775–791. doi:10.1109/TMM.2006.876282

    Article  Google Scholar 

  33. Tian Z, Xue J, Lan X, Li C, Zheng N (2014) Object segmentation and key-pose based summarization for motion video. Multimed Tools Appl 72(2):1773–1802. doi:10.1007/s11042-013-1488-7

    Article  Google Scholar 

  34. Uchihashi S, Foote J, Girgenson A, Boreczky J (1999) Video manga: generating semantically meaningful video summaries. In: Proceedings of the 7th ACM international conference on multimedia. doi:10.1145/319463.319654

  35. Wang M, Hong R, Li G, Zha Z, Yan S, Chua T (2012) Event driven web video summarization by tag localization and key-shot identification. IEEE Trans Multimed 14(4):975–985. doi:10.1109/TMM.2012.2185041

    Article  Google Scholar 

  36. Xiang X, Kankanhalli MS (2011) Affect-based adaptive presentation of home videos. In: Proceedings of the 19th ACM international conference on multimedia. doi:10.1145/2072298.2072370

  37. Yeung MM, Yeo BL (1997) Video visualization for compact presentation and fast browsing of pictorial content. IEEE Trans Circ Syst Vid Technol 7(5):771–785. doi:10.1109/76.633496

    Article  Google Scholar 

  38. Zhou H, Hermans T, Karandikar A, Rehg J (2010) Movie genre classification via scene categorization. In: Proceedings of ACM international conference on multimedia. doi:10.1145/1873951.1874068

Download references

Acknowledgments

The work described in this paper was supported by the National Natural Science Foundation of China (No. 61375016) and the Science and Technology Commission of Shanghai Municipality (No. 16511102702).

Author information

Authors and Affiliations

  1. Shanghai Key Laboratory of Multidimensional Information Processing, Dept. of Computer Science and Technology, East China Normal University, 3663 N. Zhongshan Rd., Shanghai, China

    Shan Sun, Feng Wang & Liang He

Authors
  1. Shan Sun
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Feng Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Liang He
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Feng Wang.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Sun, S., Wang, F. & He, L. Movie summarization using bullet screen comments. Multimed Tools Appl 77, 9093–9110 (2018). https://doi.org/10.1007/s11042-017-4807-6

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-017-4807-6

Keywords

Search

Navigation