Skip to main content
SpringerLink
Log in

MARS: parallelism-based metrically accurate 3D reconstruction system in real-time

  • Special Issue Paper
  • Published:
Journal of Real-Time Image Processing Aims and scope Submit manuscript

Abstract

Due to the increasing application demands, lightweight device-based 3D recovery draws many attentions from a wide group of researchers in both academic and industrial fields. The current 3D reconstruction solutions are commonly achieved either using depth data or RGB data. The depth data usually come from a deliberately designed hardware for specific tasks, while the RGB data-based solutions only employ a single RGB camera with vision-based computing algorithms. Limitations are expected from both. Depth sensors are commonly either bulky or relatively expensive compared to RGB cameras, thus of less flexibility. Normal RGB cameras usually have better mobility but less accuracy in 3D sensing than depth sensors. Recently, machine learning based depth estimation has also been presented. However, its accuracy is still limited. To improve the flexibility of the 3D reconstruction system without loss in accuracy, this paper presents a solution of unconstrained Metrically Accurate 3D Reconstruction System (MARS) for 3D sensing based on a consumer-grade camera. With a simple initialization from a depth map, the system can achieve incremental 3D reconstruction with a stable metric scale. Experiments are conducted using both real-world data and public datasets. Competitive results are obtained using the proposed system compared with several existing methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  1. Meerits, S., Nozick, V., Saito, H.: Real-time scene reconstruction and triangle mesh generation using multiple RGB-D cameras. J. Real-Time Image Process. 16(6), 2247–2259 (2017)

    Article  Google Scholar 

  2. Angladon, V., Gasparini, S., Charvillat, V., Pribanic, T., Petkovic, T., Donlic, M., Ahsan, B., Bruel, F.: An evaluation of real-time RGB-D visual odometry algorithms on mobile devices. J. Real-Time Image Process. 16(5), 1643–1660 (2019)

    Article  Google Scholar 

  3. Lv, Z., Yin, T., Zhang, X., Song, H., Chen, G.: Virtual reality smart city based on WebVRGIS. IEEE Internet Things J. 3(6), 1015–1024 (2016)

    Article  Google Scholar 

  4. Sun, Y., Song, H., Jara, A.J., Bie, R.: Internet of things and big data analytics for smart and connected communities. IEEE Access 4, 766–773 (2016)

    Article  Google Scholar 

  5. Dong, Y., Song, L., Xie, R., Zhang, W.: Real-time UHD video super-resolution and transcoding on heterogeneous hardware. J. Real-Time Image Process. 17, 2029–2045 (2019)

    Article  Google Scholar 

  6. Song, H., Brandt-Pearce, M.: Range of influence and impact of physical impairments in long-haul DWDM systems. J. Lightw. Technol. 31(6), 846–854 (2013)

    Article  Google Scholar 

  7. Song, H., Brandt-Pearce, M.: A 2-D discrete-time model of physical impairments in wavelength-division multiplexing systems. J. Lightw. Technol. 30(5), 713–726 (2012)

    Article  Google Scholar 

  8. Jiang, D., Wang, W., Shi, L., Song, H.: A compressive sensing-based approach to end-to-end network traffic reconstruction. IEEE Trans. Netw. Sci. Eng. 1 (2018)

  9. Song, H.: Digital image watermarking method based on DCT and fractal encoding. IET Image Process. 11(10), 815–821 (2017)

    Article  Google Scholar 

  10. Davison, A.J., Reid, I.D., Molton, N.D., Stasse, O.: MonoSLAM: real-time single camera SLAM. Pattern Anal. Mach. Intell. IEEE Trans. 29(6), 1052–1067 (2007)

    Article  Google Scholar 

  11. Klein, G., Murray, D., Parallel tracking and mapping for small AR workspaces. In: Mixed and Augmented Reality: ISMAR 2007. 6th IEEE and ACM International Symposium on. IEEE 2007, pp. 225–234 (2007)

  12. Zhou, T., Brown, M., Snavely, N., Lowe, D.G.: Unsupervised learning of depth and ego-motion from video. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017)

  13. He, L., Wang, G., Hu, Z.: Learning depth from single images with deep neural network embedding focal length. IEEE Trans. Image Process. 27(9), 4676–4689 (2018)

    Article  MathSciNet  Google Scholar 

  14. Rodríguez-Martín, M., Rodríguez-Gonzálvez, P., González-Aguilera, D., Fernández-Hernández, J.: Feasibility study of a structured light system applied to welding inspection based on articulated coordinate measure machine data. IEEE Sens. J. 17(13), 4217–4224 (2017)

    Article  Google Scholar 

  15. Adam, A., Dann, C., Yair, O., Mazor, S., Nowozin, S.: Bayesian time-of-flight for realtime shape, illumination and albedo. IEEE Trans. Pattern Anal. Mach. Intell. 39(5), 851–864 (2017)

    Article  Google Scholar 

  16. Rosen, J., Kelner, R.: Three-dimensional imaging by self-reference single-channel digital incoherent holography. IEEE Trans. Ind. Inform. 12(4), 1571–1583 (2016)

    Article  Google Scholar 

  17. Wei, S., Zhou, C., Wang, S., Liu, K., Fan, X., Ma, J.: Colorful 3-D imaging using an infrared Dammann grating. IEEE Trans. Ind. Inform. 12(4), 1641–1648 (2016)

    Article  Google Scholar 

  18. Zhang, S., Yu, H., Wang, T., Qi, L., Dong, J., Liu, H.: Dense 3D facial reconstruction from a single depth image in unconstrained environment. Virtual Real. 22(1), 37–46 (2018)

    Article  Google Scholar 

  19. Huang, A. S., Bachrach, A., Henry, P., Krainin, M., Maturana, D., Fox, D., Roy, N.: Visual odometry and mapping for autonomous flight using an RGB-D camera. In: Robotics Research. Springer, pp. 235–252 (2017)

  20. Dai, A., Niessner, M., Zollhöfer, M., Izadi, S., Theobalt, C.: BundleFusion: real-time globally consistent 3D reconstruction using on-the-fly surface reintegration. ACM Trans. Graph 36(3), 24:1–24:18 (2017)

    Article  Google Scholar 

  21. Choi, S., Zhou, Q.-Y., Koltun, V.: Robust reconstruction of indoor scenes. In: The IEEE conference on computer vision and pattern recognition (CVPR), pp. 5556–5565 (2015)

  22. Whelan, T., Leutenegger, S., Salas-Moreno, R. F., Glocker, B., Davison, A. J.: ElasticFusion: dense SLAM without a pose graph, robotics: science and systems (2015)

  23. Newcombe, R.A., Izadi, S., Hilliges, O., Molyneaux, D., Kim, D., Davison, A.J., Kohi, P., Shotton, J., Hodges, S., Fitzgibbon, A., KinectFusion: Real-time dense surface mapping and tracking. In: Mixed and Augmented Reality (ISMAR): 10th IEEE international Symposium on. IEEE, vol. 2011, pp. 127–136 (2011)

  24. Newcombe, R.A., Lovegrove, S. J., Davison, A.J.: DTAM: Dense tracking and mapping in real-time. In: Computer vision (ICCV), 2011 IEEE international conference on. IEEE, pp. 2320–2327 (2011)

  25. Magerand, L., Bue, A.D.: Revisiting projective structure for motion: a robust and efficient incremental solution. IEEE Trans. Pattern Anal. Mach. Intell. 42(2), 430–443 (2018)

    Article  Google Scholar 

  26. Tao, M.W., Srinivasan, P.P., Hadap, S., Rusinkiewicz, S., Malik, J., Ramamoorthi, R.: Shape estimation from shading, defocus, and correspondence using light-field angular coherence. IEEE Trans. Pattern Anal. Mach. Intell. 39(3), 546–560 (2017)

    Article  Google Scholar 

  27. Liao, J., Buchholz, B., Thiery, J.-M., Bauszat, P., Eisemann, E.: Indoor scene reconstruction using near-light photometric stereo. IEEE Trans. Image Process. 26(3), 1089–1101 (2017)

    Article  MathSciNet  Google Scholar 

  28. Tijmons, S., de Croon, G.C.H.E., Remes, B.D.W., Wagter, C.D., Mulder, M.: Obstacle avoidance strategy using onboard stereo vision on a flapping wing MAV. IEEE Trans. Robot. 33(4), 858–874 (2017)

    Article  Google Scholar 

  29. Yang, S., Scherer, S.A., Yi, X., Zell, A.: Multi-camera visual SLAM for autonomous navigation of micro aerial vehicles. Robot. Auton. Syst. 93, 116–134 (2017)

    Article  Google Scholar 

  30. Liu, F., Shen, C., Lin, G., Reid, I.: Learning depth from single monocular images using deep convolutional neural fields. IEEE Trans. Pattern Anal. Mach. Intell. 38(10), 2024–2039 (2015)

    Article  Google Scholar 

  31. Rublee, E., Rabaud, V., Konolige, K., Bradski, G.: ORB: an efficient alternative to SIFT or SURF. In: Proceedings of ICCV, IEEE, vol. 58, no. 11, pp. 2564–2571 (2011)

  32. Lepetit, V., Moreno-Noguer, F., Fua, P.: Epnp: an accurate o (n) solution to the pnp problem. Int. J. Comput. Vis. 81(2), 155–166 (2009)

    Article  Google Scholar 

  33. Kerl, C., Sturm, J., Cremers, D.: Dense visual SLAM for RGB-D cameras. In: 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 2100–2106 (2013)

  34. Endres, F., Hess, J., Engelhard, N., Sturm, J., Cremers, D., Burgard, W.: An evaluation of the RGB-D SLAM system. In: Robotics and Automation (ICRA), 2012 IEEE International Conference on. IEEE, pp. 1691–1696 (2012)

  35. Stückler, J., Behnke, S.: Multi-resolution surfel maps for efficient dense 3D modeling and tracking. J. Vis. Commun. Image Represent. 25(1), 137–147 (2014)

    Article  Google Scholar 

  36. Whelan, T., Johannsson, H., Kaess, M., Leonard, J. J., McDonald, J.: Robust real-time visual odometry for dense RGB-D mapping. In: 2013 IEEE International Conference on Robotics and Automation, pp. 5724–5731 (2013)

  37. Handa, A., Whelan, T., McDonald, J., Davison, A. J.: A benchmark for RGB-D visual odometry, 3D reconstruction and SLAM. In: 2014 IEEE International Conference on Robotics and Automation (ICRA), pp. 1524–1531 (2014)

Download references

Author information

Authors and Affiliations

  1. Ocean University of China, Qingdao, China

    Shu Zhang & Junyu Dong

  2. Shandong University of Science and Technology, Qingdao, China

    Ting Wang

  3. Wuhan University of Science and Technology, Wuhan, China

    Gongfa Li

  4. University of Portsmouth, Portsmouth, UK

    Hui Yu

Authors
  1. Shu Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ting Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Gongfa Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Junyu Dong
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Hui Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Junyu Dong or Hui Yu.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This work was supported by the Engineering and Physical Sciences Research Council (EPSRC) (EP/N025849/1); National Natural Science Foundation of China (NSFC) (41906177, 41927805, 51575407); China Postdoctoral Science Foundation Grant (2019M652476); the Fundamental Research Funds for the Central Universities, China (201964022); International Science and Technology Cooperation Program of China (ISTCP) (2014DFA10410); Shandong Provincial Natural Science Foundation, China (ZR2018ZB0852).

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, S., Wang, T., Li, G. et al. MARS: parallelism-based metrically accurate 3D reconstruction system in real-time. J Real-Time Image Proc 18, 393–405 (2021). https://doi.org/10.1007/s11554-020-01031-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11554-020-01031-5

Keywords

Search

Navigation