Elsevier

Measurement

Volume 151, February 2020, 107114
Measurement

Three-dimensional reconstruction with single-shot structured light dot pattern and analytic solutions

https://doi.org/10.1016/j.measurement.2019.107114Get rights and content

Highlights

  • We propose a 3D reconstruction approach with single-shot projection of dot pattern.

  • We determine two rays that pass through the surface point of the object.

  • We calculate the 3D coordinate of the object with analytical solutions.

  • We propose an iterative segmentation method based on the areas of the segmented dots.

  • We match the dots in two views based on their calculated indexes in the row and column directions.

Abstract

Structured light three-dimensional reconstruction is one of the most important techniques in computer vision. However, most existing methods require multiple projections of the designed patterns to achieve the closed form solution, which makes them fail to measure the dynamic objects. In this paper, we propose a two-view based single-shot structured light 3D reconstruction approach that calculates the 3D coordinates of the object with analytical solutions. The designed structured light dot pattern is projected onto the object, distorted and imaged in two cameras. For each point in the structured light dot pattern, there is a ray of light that goes through the optical center and the physically imaged point on the camera’s image plane. Two rays could be determined by two cameras and their intersection solves the unknown point on the object in the 3D world coordinate system. To segment the dot pattern robustly and efficiently, an iterative dot segmentation method is proposed. Experimental results verified the effectiveness of the proposed approach in reconstructing objects without bright colors, rich textures, large discontinuities and obvious occlusions.

Introduction

Three-dimensional imaging is a multidisciplinary cross emerging subject that has got rapid development in both theoretical research and practical applications. The popular three-dimensional techniques include time of flight [1], stereo vision [2] and structured light scanning [3], [4], [5]. The resolution and accuracy of the time of flight method are low compared to the other two methods. The computation complexity of stereo vision method is high and its accuracy is easily affected by the texture and lighting condition of the imaged object. On the other hand, structured light scanning method has achieved higher accuracy, higher resolution and low computation complexity [3], [4], [5]. The application field of three-dimensional imaging is very broad and the popular applications are quality inspection and reverse engineering in the manufacturing industries. The criteria to judge if a three-dimensional imaging technique is good include reconstruction accuracy, efficiency and single-shot. Different applications might emphasize different criteria. For some applications, high reconstruction accuracy is required and the efficiency could be neglected. Then structured light scanning is a good option for such kind of applications. In some applications, the object is moving or changing shape frequently and single-shot is required. In these situations, time of flight and stereo vision could work while traditional structured light scanning techniques [4] will not work because it requires projecting multiple designed patterns onto the static object to capture multiple images to calculate a closed form solution for the 3D shape. There are many one-view based structured light methods that are capable of single-shot 3D measurement. For instance, Fourier transform profilometry (FTP) [6], [7] is a popular single-shot structured light scanning technique that is capable of reconstructing dynamic or deformable shapes from one single image. However, it is challenging to calculate the phase map from one single image [8]. Consequently, FTP is only capable of reconstructing object with relatively simple shapes. In [9], the phase map is also calculated form a single image by clustering the distorted bright lines. When there are occlusions or large discontinuities, the clustering methods need to be modified significantly and designed specifically for different models, which might cost great effort in reconstructing some complex models. In [10], the 3D shape is reconstructed by single-shot of sparse landmarks and the dense gradient information. Different from phase-map-based methods, this method calculates the surface height by inference and estimation instead of closed form solution. In [11], single-shot reconstruction is achieved from local connection information of the projected grid pattern. In [12], the grid pattern is generated with two projectors and single-shot is achieved by calculating dense phase information. All the single-shot methods mentioned above are promising, but not perfect yet. As a result, few are applied in the practical applications.

To achieve better accuracy, many researchers have utilized the structured light stripes to improve the robustness of stereo vision [13], [14], [15]. Two views to the same region illuminated by the same structured light stripes are captured by two cameras. The depth of the point on the structured light stripes is calculated by triangulation. Since the structured light improves the accuracy of stereo matching, these two-view based single shot structured light methods have achieved better accuracy compared to stereo method and some other one-view based structured light methods [13], [14], [15]. In this paper, we also propose a two-view based single-shot structured light 3D reconstruction approach. Instead of computing the depth by triangulation [13], [14], [15], the proposed method computes the three-dimensional coordinate with analytic solutions. Instead of using the structured light stripes, the proposed method utilizes the structured light dot pattern that is also used in [16] to reconstruct the 3D specular surface. In [16], the specular surface is measured and the projected ray is reflected from the specular surface onto a beam splitter. Half ray transmits through the beam splitter and imaged on the first diffusive plane while half ray is reflected and imaged on the second diffusive plane. Each ray is determined by two imaged points on the two diffusive planes. In this paper, the diffuse surface is reconstructed. Hence, the projected dots onto the object could be imaged on the image planes of the two cameras directly. Each ray is determined by the imaged point on the image plane of the camera and the optical center of the camera. As a result, the imaging system is simplified greatly and the system calibration method is also different. The designed dot pattern is projected onto the object and the object is imaged in two cameras respectively. In each camera view, the same amount points from the structured dot pattern are calculated. The calculated points in the left camera correspond to those in the right camera one by one. For each point, pl in the left camera, there is one corresponding point, pr in the right camera and these two points correspond to one surface point, Po on the object. If we could determine the ray that passes through the corresponding point, Pl or Pr on the physical image plane of the camera and the optical center, we could obtain the intersection of two rays on the surface of the object and determine the surface point. To determine the ray, we first use stereo calibration to get the translation vector and the rotation matrix between the two cameras [17]. We assume the optical center of the left camera is at the origin of the world coordinate system (0, 0, 0). Then, the optical center of the right camera could be calculated by the rotation matrix and the translation vector. We assume the equation of the physical image plane in the left camera is at z = f. Then, the three-dimensional world coordinates of the points Pl on the left physical image plane could be obtained directly from the two-dimensional camera coordinate of the point and the plane equation. The three-dimensional world coordinate of the corresponding point Pr in the right physical image plane are calculated based on the two-dimensional camera coordinate of the point and the calculated rotation matrix and the translation vector. With two points in the world coordinate system, the ray passes through the point, Pl or Pr is determined. After the equations of these two rays are computed, the point on the surface of the object is calculated as their intersection. With enough points in the designed dot pattern, enough points on the surface of the object could be obtained. The three-dimensional shape of the object could be reconstructed.

The dots in the designed dot pattern should be distinguished from each other and be as many as possible. In either camera’s view, each dot represents a ray passing through the optical centre of the camera and the image plane. We need to match the dots in the left camera’s view and the dots in the right camera’s view automatically. To this end, all the dots need to be clustered robustly. We design the pattern in RGB format to facilitate the clustering process. Therefore, the dots in the same colour could be clustered independently and robustly based on the Euclidean distance. Before dot clustering, the RGB image needs to be segmented robustly and efficiently. We propose an iterative dot segmentation method that is significantly more robust and efficient than state of the art K-means method [19], [20], [21].

This paper is organized as follows. In Section 2, the proposed single shot three-dimensional reconstruction approach is described. In Section 3, the iterative dot segmentation method is proposed and compared with state of the art K-means method. In Section 4, Experimental results are shown to verify the effectiveness of the proposed methods. Section 5 concludes the paper.

Section snippets

The proposed single shot reconstruction approach

The proposed two-view based single shot three-dimensional imaging system is shown in Fig. 1. The structured light dot pattern is projected onto the object and two cameras captured the images synchronously. The three-dimensional shape of the object is reconstructed by processing the structured light dots in the two images.

The relationship between the point on the surface of the object and the pixel on the image plane follows the pin-hole camera imaging model. Fig. 2 illustrates the pin-hole

The proposed dot segmentation method

The reconstruction of the dynamic or deformable shape raises higher demand in image processing for both accuracy and computational efficiency. If either the accuracy or the processing speed is not high enough, the shape could not be reconstructed on line. What is worse is that the shape might not be reconstructed accurately if one critical dot is missing in the segmentation results. To meet this high demand, we propose a three-channel-combined RGB dot segmentation method for the proposed

The proposed dot matching method

To match the corresponding points in two views, we cluster the segmented points in three channels respectively. In each channel, the segmented dots are dilated with the structure element B={0,0} five times to make the adjacent dots connected and form the line image.IL=ICB=z|BszIC

where Bs denotes the symmetric or supplement of B. IC is the segmented dot image in different channels and it equals IR, IG or IB. IL is the line image. The lines in IL are clustered from top to bottom as

Experimental results

In this section, we will demonstrate the effectiveness of the proposed imaging system. We show the practically established imaging system in Fig. 11, where a EB-301MS EPSON projector is used to project the designed dot pattern onto the object and two GS3-U3-15S5C point grey cameras with Kowa 8 mm lens are used to capture the dot patterns into the imaging system. For the established system, the normal working distances are from 375 mm to 500 mm and the field of views are from 225 mm to 300 mm.

Conclusion and future work

In this paper, a two-view based three dimensional reconstruction approach is proposed to meet the single-shot requirement during measuring moving or dynamic objects. The proposed method is based on analytic solutions that could achieve precise measurement in situations where there are no noises or lens distortions. Hence, the reconstruction accuracy of the proposed approach could be improved unlimitedly. To segment the dot pattern robustly and efficiently, an iterative segmentation method based

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

References (31)

  • B. Li et al.

    Motion-induced error reduction by combining Fourier transform profilometry with phase-shifting profilometry

    Opt Express

    (2016)
  • M.D. Martino et al.

    One-shot 3D scanning by combining sparse landmarks with dense gradient information

    Opt. Lasers Eng.

    (2018)
  • H. Kawasaki et al.

    Dynamic scene reconstruction using a single structured light pattern

  • R. Sagawa et al.

    Dense one-shot 3D reconstruction by detecting continuous regions with parallel line projection

    Proc. Int. Conf. Computer Vision

    (2011)
  • J. Davis et al.

    Spacetime stereo: a unifying framework for depth from triangulation

    IEEE Trans. Pattern Anal. Mach. Intell.

    (2005)
  • Cited by (30)

    • Systematic method for evaluating the performance of three-dimensional optical scanners by structured light projection applied to ballistic vests tests

      2022, Measurement: Journal of the International Measurement Confederation
      Citation Excerpt :

      It is worth mentioning that the caliper presents itself as the preferred measuring instrument used in this context, mainly due to the low cost involved, although there are studies [31,32] that strongly suggest the use of three-dimensional digitalization technologies for replacing conventional instruments in various applications. This type of equipment is characterized by having as operating principle the projection of luminous patterns or structures on a certain object and posterior capture of images with the same patterns with some type of distortion [10,33]. Several configurations are possible, using different types of projectors and cameras that capture different images of the same object providing data so that image processing algorithms can identify the same point in different images [34].

    • Fast adaptive phase unwrapping algorithm based on improved bucket sorting

      2021, Optics and Lasers in Engineering
      Citation Excerpt :

      At the same time, it is of great practical significance to study efficient and accurate active structured light measurement technology [1-3]. Active structured light reconstruction technology is based on auxiliary light source projection for 3D reconstruction and measurement, among which the most representative is the phase measurement method [4,5]. The phase measurement method [6,7] mainly uses the multi-step phase shift method to project the grating to the measured object, and combines the phase unwrapping algorithm to obtain the surface shape of the object.

    • Shape from apparent contours for bent pipes with constant diameter under perspective projection

      2021, Measurement: Journal of the International Measurement Confederation
    View all citing articles on Scopus
    View full text