Skip to main content
SpringerLink
Log in

A No-Reference and Full-Reference image quality assessment and enhancement framework in real-time

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

These days, social media holds a large portion of our daily lives. Millions of people post their images using a social media platform. The enormous amount of images shared on social network presents serious challenges and requires massive computing resources to ensure efficient data processing. However, images are subject to a wide range of distortions in real application scenarios during the processing, transmission, sharing, or combination of many factors. So, there is a need to guarantee acceptable delivery content, even though some distorted images do not have access to their original version. In this paper, we present a framework developed to process a large amount of images in real-time while estimating and assisting in the enhancement of the No-Reference and Full-Reference image quality. Our quality evaluation is measured using a Convolutional Neural Network, which is tuned by the objective quality methods, in addition to the face alignment metric and enhanced with the help of a Super-Resolution Model. A set of experiments is conducted to evaluate our proposed approach.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  1. (2015) Apache storm—concepts. [Online]. Available: http://storm.apache.org/releases/current/Concepts.html

  2. (2015) Setting up a development environment [Online]. Available: http://storm.apache.org/releases/1.0.6/Setting-up-development-environment.html

  3. (2018) Apache storm cluster architecture. [Online]. Available: http://storm.apache.org/releases/1.0.6/Setting-up-development-environment.html

  4. Abaza A (2014) Design and evaluation of photometric image quality measures for effective face recognition. IET Biometrics 3(10):314–324. [Online]. Available: https://digital-library.theiet.org/content/journals/10.1049/iet-bmt.2014.0022

    Article  Google Scholar 

  5. Agrawal P, Narayanan PJ (2011) Person de-identification in videos. IEEE Trans Circ Syst Video Technol 21(3):299–310

    Article  Google Scholar 

  6. Aggarwal CC, Hinneburg A, Keim DA (2001) On the surprising behavior of distance metrics in high dimensional space. In: Van den Bussche J, Vianu V (eds) Database theory—ICDT 2001. Springer, Berlin, pp 420–434

  7. al Bouna B, Chbeir R, Gabillon A (2011) The image protector—a flexible security rule specification toolkit. In: SECRYPT, pp 345–350

  8. Bai X, Wang J, Simons D, Sapiro G (2009) Video snapcut: robust video object cutout using localized classifiers. In: ACM SIGGRAPH 2009 papers, ser. SIGGRAPH ’09. ACM, New York, pp 70:1–70:11

  9. Bianco S, Cadène R, Celona L, Napoletano P (2018) Benchmark analysis of representative deep neural network architectures. IEEE Access 6:64270–64277

    Article  Google Scholar 

  10. Bosse S, Maniry D, Müller K, Wiegand T, Samek W (2018) Deep neural networks for no-reference and full-reference image quality assessment. IEEE Trans Image Process 27(1):206–219

    Article  MathSciNet  Google Scholar 

  11. Caviedes J, Gurbuz S (2002) No-reference sharpness metric based on local edge kurtosis. In: Proceedings. International conference on image processing, vol 3, pp III–III

  12. Chami Z, Bouna BAL, Abou Jaoude C, Chbeir R (2019) A real-time multimedia data quality assessment framework, 270–276

  13. Chuang Y-Y, Agarwala A, Curless B, Salesin DH, Szeliski R (2002) Video matting of complex scenes. In: Proceedings of the 29th annual conference on computer graphics and interactive techniques, ser. SIGGRAPH ’02. ACM, New York, pp 243–248

  14. Ciregan D, Meier U, Schmidhuber J (2012) Multi-column deep neural networks for image classification. In: 2012 IEEE conference on computer vision and pattern recognition, pp 3642–3649

  15. De Bruyne S, De Schrijver D, De Neve W, Van Deursen D, Van de Walle R (2007) Enhanced shot-based video adaptation using mpeg-21 generic bitstream syntax schema. In: IEEE symposium on computational intelligence in image and signal processing, 2007. CIISP 2007, pp 380–385

  16. Dong C, Loy CC, He K, Tang X (2014) Image super-resolution using deep convolutional networks. IEEE Trans Pattern Anal Mach Intell 38:12

    Google Scholar 

  17. Dutta A, Veldhuis RNJ, Spreeuwers LJ (2015) Predicting face recognition performance using image quality. CoRR arXiv:1510.07119. [Online]. Available: 1510.07119

  18. El-Khoury V, Bennani N, Coquil D (2010) Utility function for semantic video content adaptation. In: Proceedings of the 12th international conference on information integration and web-based applications & services, ser. iiWAS ’10. [Online]. Available: https://doi.org/10.1145/1967486.1967649. Association for Computing Machinery, New York, pp 921–924

  19. Fan J, Luo H, Hacid M-S, Bertino E (2005) A novel approach for privacy-preserving video sharing. In: Proceedings of the 14th ACM international conference on Information and knowledge management, ser. CIKM ’05. ACM, New York, pp 609–616

  20. Gang Z, Chia L-T, Zongkai Y (2004) Mpeg-21 digital item adaptation by applying perceived motion energy to h.264 video. In: 2004 international conference on image processing, 2004. ICIP ’04, vol 4, pp 2777–2780

  21. Ghadiyaram D, Bovik AC (2016) Massive online crowdsourced study of subjective and objective picture quality. IEEE Trans Image Process 25(1):372–387

    Article  MathSciNet  Google Scholar 

  22. Gudivada VN, Raghavan VV (1995) Content based image retrieval systems. Computer 28(9):18–22

    Article  Google Scholar 

  23. He L, Tao D, Li X, Gao X (2012) Sparse representation for blind image quality assessment. In: 2012 IEEE conference on computer vision and pattern recognition, pp 1146–1153

  24. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: The IEEE conference on computer vision and pattern recognition (CVPR)

  25. Herranz L (2007) Integrating semantic analysis and scalable video coding for efficient content-based adaptation. Multimed Syst 13:103–118. https://doi.org/10.1007/s00530-007-0090-0

    Article  Google Scholar 

  26. Hou L, Yu C, Samaras D Squared earth mover’s distance-based loss for training deep neural networks. CoRR arXiv:1611.05916, 2016. [Online]. Available: 1611.05916

  27. Huynh-Thu Q (2008) Scope of validity of psnr in image/video quality assessment. Electron Lett 44(1):800–801. [Online]. Available: https://digital-library.theiet.org/content/journals/10.1049/el_20080522

    Article  Google Scholar 

  28. Kang L, Ye P, Li Y, Doermann D (2014) Convolutional neural networks for no-reference image quality assessment. In: The IEEE conference on computer vision and pattern recognition (CVPR)

  29. Kephart JO, Das R (2007) Achieving self-management via utility functions. IEEE Internet Comput 11(1):40–48

    Article  Google Scholar 

  30. Kim J, Lee S (2017) Fully deep blind image quality predictor. IEEE J Sel Top Signal Process 11(1):206–220

    Article  Google Scholar 

  31. Kim CS, Sohn H, Neve WD, Ro YM (2009) An objective perceptual quality-based adte for adapting mobile svc video content. IEICE Trans Inf Syst E92.D(1):93–96

    Article  Google Scholar 

  32. Li Q, Lin W, Fang Y (2016) No-reference quality assessment for multiply-distorted images in gradient domain. IEEE Signal Process Lett 23(4):541–545

    Article  Google Scholar 

  33. Ledig C, Theis L, Huszar F, Caballero J, Cunningham A, Acosta A, Aitken A, Tejani A, Totz J, Wang Z, Shi W (2017) Photo-realistic single image super-resolution using a generative adversarial network, pp 105–114

  34. Lian Z, Li Y, Tao J, Huang J, Niu M -Y (2019) Expression analysis based on face regions in read-world conditions. Int J Autom Comput 17

  35. Lin K-Y, Wang G (2018) Hallucinated-iqa: No-reference image quality assessment via adversarial learning. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR)

  36. Liu Z, Luo P, Wang X, Tang X (2015) Deep learning face attributes in the wild. In: Proceedings of international conference on computer vision (ICCV)

  37. Liu X, van de Weijer J, Bagdanov AD (2017) Rankiqa: learning from rankings for no-reference image quality assessment. In: Proceedings of the IEEE international conference on computer vision (ICCV)

  38. Majumdar S (2016) Image super resolution. https://github.com/titu1994/Image-Super-Resolution

  39. Marziliano P, Dufaux F, Winkler S, Ebrahimi T (2002) A no-reference perceptual blur metric. In: Proceedings. International conference on image processing, vol 3, pp III–III

  40. Meesters L, Martens J-B (2002) A single-ended blockiness measure for jpeg-coded images. Signal Process 82(3):369–387. [Online]. Available: http://www.sciencedirect.com/science/article/pii/S0165168401001773

    Article  Google Scholar 

  41. Mittal A, Moorthy AK, Bovik AC (2011) Blind/referenceless image spatial quality evaluator. In: 2011 Conference record of the forty fifth asilomar conference on signals, systems and computers (ASILOMAR), pp 723–727

  42. Moorthy AK, Bovik AC (2010) A two-step framework for constructing blind image quality indices. IEEE Signal Process Lett 17(5):513–516

    Article  Google Scholar 

  43. Moorthy AK, Bovik AC (2011) Blind image quality assessment: from natural scene statistics to perceptual quality. IEEE Trans Image Process 20 (12):3350–3364

    Article  MathSciNet  Google Scholar 

  44. Mu M, Cerqueira E, Boavida F, Mauthe A (2009) Quality of experience management framework for real-time multimedia applications. IJIPT 4:54–64

    Article  Google Scholar 

  45. Newton EM, Sweeney L, Malin B (2005) Preserving privacy by de-identifying face images. IEEE Trans Knowl Data Eng 17(2):232–243

    Article  Google Scholar 

  46. Nguyen SM, Ogino M, Asada M Real-time face swapping as a tool for understanding infant self-recognition. CoRR arXiv:1112.2095, 2011. [Online]. Available: 1112.2095

  47. Ponomarenko N, Jin L, Ieremeiev O, Lukin V, Egiazarian K, Astola J, Vozel B, Chehdi K, Carli M, Battisti F, Jay Kuo C-C (2015) Image database tid2013: peculiarities, results and perspectives. Signal Process: Image Commun 30:57–77. [Online]. Available: http://www.sciencedirect.com/science/article/pii/S0923596514001490

    Google Scholar 

  48. Prangl M, Szkaliczki T, Hellwagner H (2007) A framework for utility-based multimedia adaptation. IEEE Trans Circ Syst Video Technol 17(6):719–728

    Article  Google Scholar 

  49. Rippel O, Bourdev L (2017) Real-time adaptive image compression

  50. Rueangprathum A, Limsiroratana S, Witosurapot S (2016) User-driven multimedia adaptation framework for context-aware learning content service. J Adv Inf Technol 7:182–185

    Google Scholar 

  51. Saad MA, Bovik AC, Charrier C (2012) Blind image quality assessment: a natural scene statistics approach in the dct domain. IEEE Trans Image Process 21(8):3339–3352

    Article  MathSciNet  Google Scholar 

  52. Sheikh HR, Sabir MF, Bovik AC (2006) A statistical evaluation of recent full reference image quality assessment algorithms. IEEE Trans Image Process 15(11):3440–3451

    Article  Google Scholar 

  53. Social media statistics in 2020. https://dustinstout.com/social-media-statistics/#instagram-stats. Accessed: 24 Jan 2020

  54. Vega MT, Mocanu DC, Famaey J, Stavrou S, Liotta A (2017) Deep learning for quality assessment in live video streaming. IEEE Signal Process Lett 24(6):736–740

    Article  Google Scholar 

  55. Vijay Venkatesh M, Cheung S-CS, Zhao J (2009) Efficient object-based video inpainting. Pattern Recogn Lett 30(2):168–179

    Article  Google Scholar 

  56. Wang Z, Bovik AC, Sheikh HR, Simoncelli EP (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 13(4):600–612

    Article  Google Scholar 

  57. Wickramasuriya J, Datt M, Mehrotra S, Venkatasubramanian N (2004) Privacy protecting data collection in media spaces. In: Proceedings of the 12th annual ACM international conference on multimedia, ser. Multimedia ’04. ACM, New York, pp 48–55

  58. Yang W, Zhang X, Tian Y, Wang W, Xue J-H, Liao Q (2019) Deep learning for single image super-resolution: a brief review. IEEE Trans Multimed 21:3106–3121

    Article  Google Scholar 

  59. Ye P, Kumar J, Kang L, Doermann D (2012) Unsupervised feature learning framework for no-reference image quality assessment. In: 2012 IEEE conference on computer vision and pattern recognition, pp 1098–1105

  60. Zhang L, Zhang L, Bovik AC (2015) A feature-enriched completely blind image quality evaluator. IEEE Trans Image Process 24(8):2579–2591

    Article  MathSciNet  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Faculty of Engineering - TICKET Lab, Antonine University, Beirut, Lebanon

    Zahi Al Chami & Chady Abou Jaoude

  2. Université de Pau et des Pays de l’Adour, E2S UPPA, LIUPPA, Bayonne, France

    Zahi Al Chami & Richard Chbeir

  3. Department of Computer Science, Claude Bernard University Lyon I, Lyon, France

    Mahmoud Barhamgi

  4. Department of Management Information Systems, College of Commerce and Business Administration, Dhofar University, Salalah, Oman

    Mansour Naser Alraja

Authors
  1. Zahi Al Chami
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Chady Abou Jaoude
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Richard Chbeir
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Mahmoud Barhamgi
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Mansour Naser Alraja
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zahi Al Chami.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This work is jointly funded from the National Council for Scientific Research in Lebanon (CNRS-L), the Antonine University, and the Agence universitaire de la Francophonie (AUF).

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Chami, Z.A., Jaoude, C.A., Chbeir, R. et al. A No-Reference and Full-Reference image quality assessment and enhancement framework in real-time. Multimed Tools Appl 81, 32491–32517 (2022). https://doi.org/10.1007/s11042-022-12334-z

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-022-12334-z

Keywords

Search

Navigation