Skip to main content
SpringerLink
Log in

LS3D: Single-View Gestalt 3D Surface Reconstruction from Manhattan Line Segments

  • Conference paper
  • First Online:
Computer Vision – ACCV 2018 (ACCV 2018)

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 11364))

Included in the following conference series:

  • 1850 Accesses

Abstract

Recent deep learning algorithms for single-view 3D reconstruction recover rough 3D layout but fail to capture the crisp linear structures that grace our urban landscape. Here we show that for the particular problem of 3D Manhattan building reconstruction, the explicit application of linear perspective and Manhattan constraints within a classical constructive perceptual organization framework allows accurate and meaningful reconstructions to be computed. The proposed Line-Segment-to-3D (LS3D) algorithm computes a hierarchical representation through repeated application of the Gestalt principle of proximity. Edges are first organized into line segments, and the subset that conforms to a Manhattan frame is extracted. Optimal bipartite grouping of orthogonal line segments by proximity minimizes the total gap and generates a set of Manhattan spanning trees, each of which is then lifted to 3D. For each 3D Manhattan tree we identify the complete set of 3D 3-junctions and 3-paths, and show that each defines a unique minimal spanning cuboid. The cuboids generated by each Manhattan tree together define a solid model and the visible surface for that tree. The relative depths of these solid models are determined by an L1 minimization that is again rooted in a principle of proximity in both depth and image dimensions. The method has relatively fewer parameters and requires no training. For quantitative evaluation, we introduce a new 3D Manhattan building dataset (3DBM). We find that the proposed LS3D method generates 3D reconstructions that are both qualitatively and quantitatively superior to reconstructions produced by state-of-the-art deep learning approaches.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    We use a minimum segment length of 100 pixels, and maximum gap between co-linear segments of 300 pixels. Sensitivity to these threshold is studied in Sect. 5.

  2. 2.

    We use a threshold distance of 100 pixels - sensitivity to this threshold is studied in Sect. 5.

References

  1. Coughlan, J.M., Yuille, A.L.: Manhattan world: orientation and outlier detection by Bayesian inference. Neural Comput. 15, 1063–1088 (2003)

    Article  Google Scholar 

  2. Kubovy, M., Wagemans, J.: Grouping by proximity and multistability in dot lattices: a quantitative Gestalt theory. Psychol. Sci. 6, 225–234 (1995)

    Article  Google Scholar 

  3. Kubovy, M., Holcombe, A.O., Wagemans, J.: On the lawfulness of grouping by proximity. Cogn. Psychol. 35, 71–98 (1998)

    Article  Google Scholar 

  4. Elder, J.H., Goldberg, R.M.: Ecological statistics of Gestalt laws for the perceptual organization of contours. J. Vis. 2, 324–353 (2002)

    Article  Google Scholar 

  5. Wagemans, J., et al.: A century of Gestalt psychology in visual perception: I. Perceptual grouping and figure-ground organization. Psychol. Bull. 138, 1172 (2012)

    Article  Google Scholar 

  6. Gupta, A., Efros, A.A., Hebert, M.: Blocks world revisited: image understanding using qualitative geometry and mechanics. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010. LNCS, vol. 6314, pp. 482–496. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-15561-1_35

    Chapter  Google Scholar 

  7. Roberts, L.G.: Machine perception of three-dimensional solids. Ph.D. thesis, Massachusetts Institute of Technology (1963)

    Google Scholar 

  8. Guzman, A.: Computer recognition of three-dimensional objects in a visual scene. Ph.D. thesis, MIT (1968)

    Google Scholar 

  9. Waltz, D.L.: Generating semantic descriptions from drawings of scenes with shadows. Technical Report AITR-271, MIT (1972)

    Google Scholar 

  10. Kanade, T.: A theory of Origami world. Artif. Intell. 13, 279–311 (1980)

    Article  MathSciNet  Google Scholar 

  11. Sugihara, K.: Machine Interpretation of Line Drawings, vol. 1. MIT Press, Cambridge (1986)

    Google Scholar 

  12. Hoiem, D., Efros, A.A., Hebert, M.: Recovering surface layout from an image. Int. J. Comput. Vis. 75, 151–172 (2007)

    Article  Google Scholar 

  13. Barinova, O., Konushin, V., Yakubenko, A., Lee, K.C., Lim, H., Konushin, A.: Fast automatic single-view 3-D reconstruction of urban scenes. In: Forsyth, D., Torr, P., Zisserman, A. (eds.) ECCV 2008. LNCS, vol. 5303, pp. 100–113. Springer, Heidelberg (2008). https://doi.org/10.1007/978-3-540-88688-4_8

    Chapter  Google Scholar 

  14. Haines, O., Calway, A.: Recognising planes in a single image. IEEE TPAMI 37, 1849–1861 (2015)

    Article  Google Scholar 

  15. Coughlan, J.M., Yuille, A.L.: Manhattan world: compass direction from a single image by Bayesian inference. In: CVPR, vol. 2, pp. 941–947 (1999)

    Google Scholar 

  16. Denis, P., Elder, J.H., Estrada, F.J.: Efficient edge-based methods for estimating Manhattan frames in urban imagery. In: Forsyth, D., Torr, P., Zisserman, A. (eds.) ECCV 2008. LNCS, vol. 5303, pp. 197–210. Springer, Heidelberg (2008). https://doi.org/10.1007/978-3-540-88688-4_15

    Chapter  Google Scholar 

  17. Tal, R., Elder, J.H.: An accurate method for line detection and Manhattan frame estimation. In: Park, J.-I., Kim, J. (eds.) ACCV 2012. LNCS, vol. 7729, pp. 580–593. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-37484-5_47

    Chapter  Google Scholar 

  18. Delage, E., Lee, H., Ng, A.Y.: Automatic single-image 3D reconstructions of indoor Manhattan world scenes. In: Thrun, S., Brooks, R., Durrant-Whyte, H. (eds.) Robotics Research. STAR, vol. 28, pp. 305–321. Springer, Heidelberg (2007). https://doi.org/10.1007/978-3-540-48113-3_28

    Chapter  Google Scholar 

  19. Hedau, V., Hoiem, D., Forsyth, D.: Recovering the spatial layout of cluttered rooms. In: ICCV, pp. 1849–1856 (2009)

    Google Scholar 

  20. Gupta, A., Hebert, M., Kanade, T., Blei, D.M.: Estimating spatial layout of rooms using volumetric reasoning about objects and surfaces. In: Lafferty, J.D., Williams, C.K.I., Shawe-Taylor, J., Zemel, R.S., Culotta, A. (eds.) NIPS. Curran Associates, Inc. (2010)

    Google Scholar 

  21. Ramalingam, S., Pillai, J.K., Jain, A., Taguchi, Y.: Manhattan junction catalogue for spatial reasoning of indoor scenes. In: CVPR 2013, pp. 3065–3072 (2013)

    Google Scholar 

  22. Mallya, A., Lazebnik, S.: Learning informative edge maps for indoor scene layout prediction. In: ICCV, pp. 936–944 (2015)

    Google Scholar 

  23. Pero, L.D., Bowdish, J., Fried, D., Kermgard, B., Hartley, E., Barnard, K.: Bayesian geometric modeling of indoor scenes. In: CVPR, pp. 2719–2726 (2012)

    Google Scholar 

  24. Felzenszwalb, P.F., Veksler, O.: Tiered scene labeling with dynamic programming. In: CVPR, pp. 3097–3104 (2010)

    Google Scholar 

  25. Schwing, A.G., Urtasun, R.: Efficient exact inference for 3D indoor scene understanding. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012. LNCS, vol. 7577, pp. 299–313. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-33783-3_22

    Chapter  Google Scholar 

  26. Yang, H., Zhang, H.: Efficient 3D room shape recovery from a single panorama. In: CVPR, pp. 5422–5430 (2016)

    Google Scholar 

  27. Dasgupta, S., Fang, K., Chen, K., Savarese, S.: Delay: robust spatial layout estimation for cluttered indoor scenes. In: CVPR, pp. 616–624 (2016)

    Google Scholar 

  28. Ramalingam, S., Brand, M.: Lifting 3D Manhattan lines from a single image. In: ICCV, pp. 497–504 (2013)

    Google Scholar 

  29. Kushal, A., Seitz, S.M.: Single view reconstruction of piecewise swept surfaces. In: 3DV, pp. 239–246 (2013)

    Google Scholar 

  30. Saxena, A., Sun, M., Ng, A.Y.: Make3D: learning 3D scene structure from a single still image. IEEE TPAMI 31, 824–840 (2009)

    Article  Google Scholar 

  31. Eigen, D., Fergus, R.: Predicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture. In: CVPR, pp. 2650–2658 (2015)

    Google Scholar 

  32. Liu, F., Shen, C., Lin, G., Reid, I.: Learning depth from single monocular images using deep convolutional neural fields. IEEE TPAMI 38, 2024–2039 (2016)

    Article  Google Scholar 

  33. Laina, I., Rupprecht, C., Belagiannis, V., Tombari, F., Navab, N.: Deeper depth prediction with fully convolutional residual networks. In: 3DV, pp. 239–248 (2016)

    Google Scholar 

  34. Liu, F., Shen, C., Lin, G.: Deep convolutional neural fields for depth estimation from a single image. In: CVPR, pp. 5162–5170 (2015)

    Google Scholar 

  35. Zhuo, W., Salzmann, M., He, X., Liu, M.: 3D box proposals from a single monocular image of an indoor scene. In: AAAI (2018)

    Google Scholar 

  36. Fu, H., Gong, M., Wang, C., Batmanghelich, K., Tao, D.: Deep ordinal regression network for monocular depth estimation. In: CVPR (2018)

    Google Scholar 

  37. Xu, D., Ouyang, W., Wang, X., Sebe, N.: PAD-Net: multi-tasks guided prediction-and-distillation network for simultaneous depth estimation and scene parsing. In: CVPR (2018)

    Google Scholar 

  38. Qi, X., Liao, R., Liu, Z., Urtasun, R., Jia, J.: GeoNet: geometric neural network for joint depth and surface normal estimation. In: CVPR (2018)

    Google Scholar 

  39. Li, Z., Snavely, N.: MegaDepth: learning single-view depth prediction from internet photos. In: CVPR (2018)

    Google Scholar 

  40. Geiger, A., Lenz, P., Stiller, C., Urtasun, R.: Vision meets robotics: the KITTI dataset. Int. J. Rob. Res. 32, 1231–1237 (2013)

    Article  Google Scholar 

  41. Garg, R., B.G., V.K., Carneiro, G., Reid, I.: Unsupervised CNN for single view depth estimation: geometry to the rescue. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9912, pp. 740–756. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46484-8_45

    Chapter  Google Scholar 

  42. Zhou, T., Brown, M., Snavely, N., Lowe, D.G.: Unsupervised learning of depth and ego-motion from video. In: CVPR (2017)

    Google Scholar 

  43. Izadinia, H., Shan, Q., Seitz, S.M.: IM2CAD. In: CVPR, pp. 2422–2431. IEEE (2017)

    Google Scholar 

  44. Almazan, E.J., Tal, R., Qian, Y., Elder, J.H.: MCMLSD: a dynamic programming approach to line segment detection. In: CVPR (2017)

    Google Scholar 

  45. Lee, D., Hebert, M., Kanade, T.: Geometric reasoning for single image structure recovery. In: CVPR, pp. 2136–2143. IEEE (2009)

    Google Scholar 

  46. Munkres, J.: Algorithms for the assignment and transportation problems. J. Soc. Ind. Appl. Math. 5, 32–38 (1957)

    Article  MathSciNet  Google Scholar 

  47. Cignoni, P., Callieri, M., Corsini, M., Dellepiane, M., Ganovelli, F., Ranzuglia, G.: MeshLab: an open-source mesh processing tool. In: Eurographics Italian Chapter Conference (2008)

    Google Scholar 

  48. Eigen, D., Puhrsch, C., Fergus, R.: Depth map prediction from a single image using a multi-scale deep network. In: NIPS, pp. 2366–2374 (2014)

    Google Scholar 

  49. Liu, C., Yang, J., Ceylan, D., Yumer, E., Furukawa, Y.: PlaneNet: piece-wise planar reconstruction from a single RGB image. In: CVPR, pp. 2579–2588 (2018)

    Google Scholar 

Download references

Acknowledgements

This research was supported by the NSERC Discovery program and the NSERC CREATE Training Program in Data Analytics & Visualization, the Ontario Research Fund, and the York University VISTA and Research Chair programs.

Author information

Authors and Affiliations

  1. York University, Toronto, Canada

    Yiming Qian & James H. Elder

  2. University of Utah, Salt Lake City, USA

    Srikumar Ramalingam

Authors
  1. Yiming Qian
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Srikumar Ramalingam
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. James H. Elder
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to James H. Elder .

Editor information

Editors and Affiliations

  1. IIIT Hyderabad, Hyderabad, India

    C.V. Jawahar

  2. ANU, Canberra, ACT, Australia

    Hongdong Li

  3. Simon Fraser University, Burnaby, BC, Canada

    Greg Mori

  4. ETH Zurich, Zurich, Zürich, Switzerland

    Konrad Schindler

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Qian, Y., Ramalingam, S., Elder, J.H. (2019). LS3D: Single-View Gestalt 3D Surface Reconstruction from Manhattan Line Segments. In: Jawahar, C., Li, H., Mori, G., Schindler, K. (eds) Computer Vision – ACCV 2018. ACCV 2018. Lecture Notes in Computer Science(), vol 11364. Springer, Cham. https://doi.org/10.1007/978-3-030-20870-7_25

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-20870-7_25

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-20869-1

  • Online ISBN: 978-3-030-20870-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation