Skip to main content
SpringerLink
Log in

Multi-camera platform for panoramic real-time HDR video construction and rendering

  • Special Issue Paper
  • Published:
Journal of Real-Time Image Processing Aims and scope Submit manuscript

Abstract

High dynamic range (HDR) images are usually obtained by capturing several images of the scene at different exposures. Previous HDR video techniques adopted the same principle by stacking HDR frames in time domain. We designed a new multi-camera platform which is able to construct and render HDR panoramic video in real time, with \(1{,}024 \times 256\) resolution and a frame rate of 25 fps. We exploit the overlapping fields of view between the cameras with different exposures to create an HDR radiance map. We propose a method for HDR frame reconstruction which merges the previous HDR imaging techniques with the algorithms for panorama reconstruction. The developed FPGA-based processing system is able to reconstruct the HDR frame using the proposed method and tone map the resulting image using a hardware-adapted global operator. The measured throughput of the system is 245 MB/s, which is, up to our knowledge, among the fastest HDR video processing system.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Chander, G., Markham, B.L., Helder, D.L.: Summary of current radiometric calibration coefficients for Landsat MSS, TM, ETM+, and EO-1 ALI sensors. Remote Sens. Environ. 113(5), 893–903 (2009)

    Article  Google Scholar 

  2. Jungmann, J.H., MacAleese, L., Visser, J., Vrakking, M.J.J., Heeren, R.M.A.: High dynamic range bio-molecular ion microscopy with the timepix detector. Anal. Chem. 83(20), 7888–7894 (2011)

    Article  Google Scholar 

  3. Bloch, C.: The HDRI handbook 2.0: high dynamic range imaging for photographers and CG artists. Rocky Nook, USA (2013)

    Google Scholar 

  4. Mann, S., Picard, R.W.: On being ’Undigital’ with digital cameras: extending dynamic range by combining differently exposed pictures. In: Proceedings of IS&T, pp. 442–448 (1995)

  5. Debevec, P.E., Malik, J.: Recovering high dynamic range radiance maps from photographs. In: ACM SIGGRAPH 97. New York, pp. 369–378 (1997)

  6. Ward, G.: Graphics gems II. Real pixels, pp. 80–83. Academic Press, San Diego (1991)

    Book  Google Scholar 

  7. Mitsunaga, T., Nayar, S.: Radiometric self calibration. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), vol. 1, pp. 374–380 (1999)

  8. Pattanaik, S.N., Reinhard, E., Ward, G., Debevec, P.E.: High dynamic range imaging—acquisition, display, and image-based lighting. Morgan Kaufmann, Massachusetts (2005)

    Google Scholar 

  9. Robertson, M.A., Borman, S., Stevenson, R.L.: Estimation-theoretic approach to dynamic range enhancement using multiple exposures. J. Electron. Imaging 12(2), 219–228 (2003)

    Article  Google Scholar 

  10. Granados, M., Ajdin, B., Wand, M., Theobalt, C., Seidel, H.P., Lensch, H.P.A.: Optimal HDR reconstruction with linear digital cameras. In: Proceedings of 23rd IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 215–222 (2010)

  11. Hasinoff, S.W., Durand, F., Freeman, W.T.: Noise-optimal capture for high dynamic range photography. In: Proceedings of 23rd IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 553–560 (2010)

  12. Mertens, T., Kautz, J., Van Reeth, F.: Exposure fusion. In: Pacific Conferece on Computer Graphics and Applications, pp. 382–390. doi:10.1109/PG.2007.17 (2007)

  13. Saleem, A., Beghdadi, A., Boashash, B.: Image fusion-based contrast enhancement. EURASIP J. Image Video Process. 2012, 10 (2012). doi:10.1186/1687-5281-2012-10

    Article  Google Scholar 

  14. Martinez-Sanchez, A., Fernandez, C., Navarro, P.J., Iborra, A.: A novel method to increase LinLog CMOS sensors’ performance in high dynamic range scenarios. Sensors 11(9), 8412–8429 (2011). doi:10.3390/s110908412

    Article  Google Scholar 

  15. Ward, G., Rushmeier, H., Piatko, C.: A visibility matching tone reproduction operator for high dynamic range scenes. IEEE Trans. Vis. Comput. Graph. 3(4), 291–306 (1997). doi:10.1109/2945.646233

    Article  Google Scholar 

  16. Pattanaik, S.N., Tumblin, J., Yee, H., Greenberg, D.P.: Time-dependent visual adaptation for fast realistic image display. In: Proceedings of ACM SIGGRAPH 00. New York, pp. 47–54. doi:10.1145/344779.344810 (2000)

  17. Drago, F., Myszkowski, K., Annen, T., Chiba, N.: Adaptive logarithmic mapping for displaying high contrast scenes. Comput. Graph. forum 22(3), 419–426 (2003). doi:10.1111/1467-8659.00689

    Article  Google Scholar 

  18. Mantiuk, R., Daly, S., Kerofsky, L.: Display adaptive tone mapping. ACM Trans. Graph. 27(3), 68:1–68:10 (2008)

    Article  Google Scholar 

  19. Reinhard, E., Stark, M., Shirley, P., Ferwerda, J.: Photographic tone reproduction for digital images. ACM Trans. Graph. 21(3), 267–276 (2002). doi:10.1145/566654.566575

    Article  Google Scholar 

  20. Fattal, R., Lischinski, D., Werman, M.: Gradient domain high dynamic range compression. ACM Trans. Graph. 21(3), 249–256 (2002). doi:10.1145/566654.566573

    Article  Google Scholar 

  21. Durand, F., Dorsey, J.: Fast bilateral filtering for the display of high-dynamic-range images. ACM Trans. Graph. 21(3), 257–266 (2002). doi:10.1145/566654.566574

    Article  Google Scholar 

  22. Kang, S.B., Uyttendaele, M., Winder, S., Szeliski, R.: High dynamic range video. ACM Trans. Graph. 22(3), 319–325 (2003). doi:10.1145/882262.882270

    Article  Google Scholar 

  23. Kalantari, N.K., Shechtman, E., Barnes, C., Darabi, S., Goldman, D.B., Sen, P.: Patch-based high dynamic range video. In: ACM Transactions on Graphics (TOG) (Proceedings of SIGGRAPH Asia 2013), vol. 32(6) (2013)

  24. Gupta, M., Iso, D., Nayar, S.: Fibonacci exposure bracketing for high dynamic range imaging. In: IEEE International Conference on Computer Vision (ICCV) (2013)

  25. Tocci, M.D., Kiser, C., Tocci, N., Sen, P.: A versatile HDR video production system. ACM Trans. Graph. 30(4), 41:1–41:10 (2011). doi:10.1145/2010324.1964936

    Article  Google Scholar 

  26. Kronander, J., Gustavson, S., Bonnet, G., Unger, J.: Unified HDR reconstruction from raw CFA data. In: Proceedings of IEEE International Conference on Computational Photography (2013)

  27. Ramachandra, V., Zwicker, M., Nguyen, T.: HDR imaging from differently exposed multiview videos. In: IEEE 3DTV Conference: The True Vision-Capture, Transmission and Display of 3D Video, pp. 85–88 (2008)

  28. Portz, T., Zhang, L., Jiang, H.: Random coded sampling for high-speed HDR video. In: IEEE International Conference on Computational Photography (ICCP). doi:10.1109/ICCPhot.6528308 (2013)

  29. Hassan, F., Carletta, J.E.: An FPGA-based architecture for a local tone-mapping operator. J. Real Time Image Process. 2(4), 293–308 (2007). doi:10.1007/s11554-007-0056-7

    Article  Google Scholar 

  30. Vytla, L., Hassan, F., Carletta, J.: A real-time implementation of gradient domain high dynamic range compression using a local Poisson solver. J. Real Time Image Process. 8(2), 153–167 (2013). doi:10.1007/s11554-011-0198-5

    Article  Google Scholar 

  31. Lapray, P.J., Heyrman, B., Rosse, M., Ginhac, D.: HDR-ARtiSt: high dynamic range advanced real-time imaging system. In: IEEE International Symposium on Circuits and Systems, pp. 1428–1431. doi:10.1109/ISCAS.2012.6271513 (2012)

  32. Lapray, P.J., Heyrman, B., Ginhac, D.: HDR-ARtiSt: an adaptive real-time smart camera for high dynamic range imaging. J. Real Time Image Process. 1–16. doi:10.1007/s11554-013-0393-7 (2014)

  33. Akyuz, A.O.: High dynamic range imaging pipeline on the GPU. J. Real Time Image Process. 1–15. doi:10.1007/s11554-012-0270-9 (2012)

  34. Akil, M., Grandpierre, T., Perroton, L.: Real-time dynamic tone-mapping operator on GPU. J. Real Time Image Process. 7(3), 165–172 (2012). doi:10.1007/s11554-011-0196-7

    Article  Google Scholar 

  35. Slomp, M., Oliveira, M.M.: Real-time photographic local tone reproduction using summed-area tables, pp. 82–91. Computer Graphics International, Istanbul (2008)

    Google Scholar 

  36. Afshari, H.: A real-time multi-aperture omnidirectional visual sensor with interconnected network of smart cameras. Ph.D. thesis, Ecole Polytechnique Federale de Lausanne (EPFL), Switzerland. doi:10.5075/epfl-thesis-5717 (2013)

  37. Brown, M., Lowe, D.: Automatic panoramic image stitching using invariant features. Int. J. Comput. Vis. 74(1), 59–73 (2007)

    Article  Google Scholar 

  38. Szeliski, R., Uyttendaele, M., Steedly, D.: Fast poisson blending using multi-splines. In: IEEE International Conference on Computational Photography (ICCP). doi:10.1109/ICCPHOT.2011.5753119 (2011)

  39. Wilburn, B., Joshi, N., Vaish, V., Talvala, E.V., Antunez, E., Barth, A., Adams, A., Horowitz, M., Levoy, M.: High performance imaging using large camera arrays. ACM trans. Graph. 24, 765–776 (2005). doi:10.1145/1073204.1073259

    Article  Google Scholar 

  40. Popovic, V., Afshari, H., Schmid, A., Leblebici, Y.: Real-time implementation of gaussian image blending in a spherical light field camera. In: Proceedings of IEEE International Conference on Industrial Technology, pp. 1173–1178. doi:10.1109/ICIT.2013.6505839 (2013)

  41. Popovic, V., Seyid, K., Akin, A., Cogal, O., Afshari, H., Schmid, A., Leblebici, Y.: Image blending in a high frame rate FPGA-based multi-camera system. J. Signal Process. Syst. 1–16. doi:10.1007/s11265-013-0858-8 (2013)

  42. Hartley, R.I., Zisserman, A.: Multiple view geometry in computer vision, 2nd edn. Cambridge University Press, Cambridge (2004)

    Book  MATH  Google Scholar 

  43. de Berg, M., van Kreveld, M., Overmars, M., Schwarzkopf, O.: Computational geometry: algorithms and applications, 2nd edn. Springer, Berlin (2000)

    Book  MATH  Google Scholar 

  44. Yoshida, A., Blanz, V., Myszkowski, K., Seidel, H.P.: Perceptual evaluation of tone mapping operators with real-world scenes. In: SPIE Human Vision & Electronic Imaging, vol. X, pp. 192–203. doi:10.1117/12.587782 (2005)

  45. Meyer-Baese, U.: Digital signal processing with field programmable gate arrays, 3rd edn. Springer, Berlin (2007)

    MATH  Google Scholar 

  46. Popovic, V., Seyid, K., Schmid, A., Leblebici, Y.: Real-time hardware implementation of multi-resolution image blending. In: IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 2741–2745. doi:10.1109/ICASSP.2013.6638155 (2013)

Download references

Acknowledgments

The authors would like to thank H. Afshari, S. Hauser and, P. Bruehlmeier for their work on designing the hardware platform.

Author information

Authors and Affiliations

  1. Microelectronic Systems Laboratory, Ecole Polytechnique Fédérale de Lausanne (EPFL), Station 11, 1015, Lausanne, Switzerland

    Vladan Popovic, Kerem Seyid, Eliéva Pignat, Ömer Çogal & Yusuf Leblebici

Authors
  1. Vladan Popovic
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Kerem Seyid
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Eliéva Pignat
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ömer Çogal
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Yusuf Leblebici
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Vladan Popovic.

Additional information

This work has been partially funded by the Science and Technology Division of the Swiss Federal Competence Center Armasuisse. The authors gratefully acknowledge the support of Xilinx, Inc., through the Xilinx University Program.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (MPEG 22882 kb)

Supplementary material 2 (MPEG 17514 kb)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Popovic, V., Seyid, K., Pignat, E. et al. Multi-camera platform for panoramic real-time HDR video construction and rendering. J Real-Time Image Proc 12, 697–708 (2016). https://doi.org/10.1007/s11554-014-0444-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11554-014-0444-8

Keywords

Search

Navigation