Three-dimensional reconstruction with single-shot structured light dot pattern and analytic solutions
Introduction
Three-dimensional imaging is a multidisciplinary cross emerging subject that has got rapid development in both theoretical research and practical applications. The popular three-dimensional techniques include time of flight [1], stereo vision [2] and structured light scanning [3], [4], [5]. The resolution and accuracy of the time of flight method are low compared to the other two methods. The computation complexity of stereo vision method is high and its accuracy is easily affected by the texture and lighting condition of the imaged object. On the other hand, structured light scanning method has achieved higher accuracy, higher resolution and low computation complexity [3], [4], [5]. The application field of three-dimensional imaging is very broad and the popular applications are quality inspection and reverse engineering in the manufacturing industries. The criteria to judge if a three-dimensional imaging technique is good include reconstruction accuracy, efficiency and single-shot. Different applications might emphasize different criteria. For some applications, high reconstruction accuracy is required and the efficiency could be neglected. Then structured light scanning is a good option for such kind of applications. In some applications, the object is moving or changing shape frequently and single-shot is required. In these situations, time of flight and stereo vision could work while traditional structured light scanning techniques [4] will not work because it requires projecting multiple designed patterns onto the static object to capture multiple images to calculate a closed form solution for the 3D shape. There are many one-view based structured light methods that are capable of single-shot 3D measurement. For instance, Fourier transform profilometry (FTP) [6], [7] is a popular single-shot structured light scanning technique that is capable of reconstructing dynamic or deformable shapes from one single image. However, it is challenging to calculate the phase map from one single image [8]. Consequently, FTP is only capable of reconstructing object with relatively simple shapes. In [9], the phase map is also calculated form a single image by clustering the distorted bright lines. When there are occlusions or large discontinuities, the clustering methods need to be modified significantly and designed specifically for different models, which might cost great effort in reconstructing some complex models. In [10], the 3D shape is reconstructed by single-shot of sparse landmarks and the dense gradient information. Different from phase-map-based methods, this method calculates the surface height by inference and estimation instead of closed form solution. In [11], single-shot reconstruction is achieved from local connection information of the projected grid pattern. In [12], the grid pattern is generated with two projectors and single-shot is achieved by calculating dense phase information. All the single-shot methods mentioned above are promising, but not perfect yet. As a result, few are applied in the practical applications.
To achieve better accuracy, many researchers have utilized the structured light stripes to improve the robustness of stereo vision [13], [14], [15]. Two views to the same region illuminated by the same structured light stripes are captured by two cameras. The depth of the point on the structured light stripes is calculated by triangulation. Since the structured light improves the accuracy of stereo matching, these two-view based single shot structured light methods have achieved better accuracy compared to stereo method and some other one-view based structured light methods [13], [14], [15]. In this paper, we also propose a two-view based single-shot structured light 3D reconstruction approach. Instead of computing the depth by triangulation [13], [14], [15], the proposed method computes the three-dimensional coordinate with analytic solutions. Instead of using the structured light stripes, the proposed method utilizes the structured light dot pattern that is also used in [16] to reconstruct the 3D specular surface. In [16], the specular surface is measured and the projected ray is reflected from the specular surface onto a beam splitter. Half ray transmits through the beam splitter and imaged on the first diffusive plane while half ray is reflected and imaged on the second diffusive plane. Each ray is determined by two imaged points on the two diffusive planes. In this paper, the diffuse surface is reconstructed. Hence, the projected dots onto the object could be imaged on the image planes of the two cameras directly. Each ray is determined by the imaged point on the image plane of the camera and the optical center of the camera. As a result, the imaging system is simplified greatly and the system calibration method is also different. The designed dot pattern is projected onto the object and the object is imaged in two cameras respectively. In each camera view, the same amount points from the structured dot pattern are calculated. The calculated points in the left camera correspond to those in the right camera one by one. For each point, pl in the left camera, there is one corresponding point, pr in the right camera and these two points correspond to one surface point, Po on the object. If we could determine the ray that passes through the corresponding point, Pl or Pr on the physical image plane of the camera and the optical center, we could obtain the intersection of two rays on the surface of the object and determine the surface point. To determine the ray, we first use stereo calibration to get the translation vector and the rotation matrix between the two cameras [17]. We assume the optical center of the left camera is at the origin of the world coordinate system (0, 0, 0). Then, the optical center of the right camera could be calculated by the rotation matrix and the translation vector. We assume the equation of the physical image plane in the left camera is at z = f. Then, the three-dimensional world coordinates of the points Pl on the left physical image plane could be obtained directly from the two-dimensional camera coordinate of the point and the plane equation. The three-dimensional world coordinate of the corresponding point Pr in the right physical image plane are calculated based on the two-dimensional camera coordinate of the point and the calculated rotation matrix and the translation vector. With two points in the world coordinate system, the ray passes through the point, Pl or Pr is determined. After the equations of these two rays are computed, the point on the surface of the object is calculated as their intersection. With enough points in the designed dot pattern, enough points on the surface of the object could be obtained. The three-dimensional shape of the object could be reconstructed.
The dots in the designed dot pattern should be distinguished from each other and be as many as possible. In either camera’s view, each dot represents a ray passing through the optical centre of the camera and the image plane. We need to match the dots in the left camera’s view and the dots in the right camera’s view automatically. To this end, all the dots need to be clustered robustly. We design the pattern in RGB format to facilitate the clustering process. Therefore, the dots in the same colour could be clustered independently and robustly based on the Euclidean distance. Before dot clustering, the RGB image needs to be segmented robustly and efficiently. We propose an iterative dot segmentation method that is significantly more robust and efficient than state of the art K-means method [19], [20], [21].
This paper is organized as follows. In Section 2, the proposed single shot three-dimensional reconstruction approach is described. In Section 3, the iterative dot segmentation method is proposed and compared with state of the art K-means method. In Section 4, Experimental results are shown to verify the effectiveness of the proposed methods. Section 5 concludes the paper.
Section snippets
The proposed single shot reconstruction approach
The proposed two-view based single shot three-dimensional imaging system is shown in Fig. 1. The structured light dot pattern is projected onto the object and two cameras captured the images synchronously. The three-dimensional shape of the object is reconstructed by processing the structured light dots in the two images.
The relationship between the point on the surface of the object and the pixel on the image plane follows the pin-hole camera imaging model. Fig. 2 illustrates the pin-hole
The proposed dot segmentation method
The reconstruction of the dynamic or deformable shape raises higher demand in image processing for both accuracy and computational efficiency. If either the accuracy or the processing speed is not high enough, the shape could not be reconstructed on line. What is worse is that the shape might not be reconstructed accurately if one critical dot is missing in the segmentation results. To meet this high demand, we propose a three-channel-combined RGB dot segmentation method for the proposed
The proposed dot matching method
To match the corresponding points in two views, we cluster the segmented points in three channels respectively. In each channel, the segmented dots are dilated with the structure element five times to make the adjacent dots connected and form the line image.
where denotes the symmetric or supplement of . is the segmented dot image in different channels and it equals , or . is the line image. The lines in are clustered from top to bottom as
Experimental results
In this section, we will demonstrate the effectiveness of the proposed imaging system. We show the practically established imaging system in Fig. 11, where a EB-301MS EPSON projector is used to project the designed dot pattern onto the object and two GS3-U3-15S5C point grey cameras with Kowa 8 mm lens are used to capture the dot patterns into the imaging system. For the established system, the normal working distances are from 375 mm to 500 mm and the field of views are from 225 mm to 300 mm.
Conclusion and future work
In this paper, a two-view based three dimensional reconstruction approach is proposed to meet the single-shot requirement during measuring moving or dynamic objects. The proposed method is based on analytic solutions that could achieve precise measurement in situations where there are no noises or lens distortions. Hence, the reconstruction accuracy of the proposed approach could be improved unlimitedly. To segment the dot pattern robustly and efficiently, an iterative segmentation method based
Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
References (31)
Phase-measurement interferometry techniques
Prog. Opt.
(1988)- et al.
Fourier transform profilometry: a review
Opt. Lasers Eng.
(2001) - et al.
Single shot three-dimensional reconstruction based on structured light line pattern
Opt. Lasers Eng.
(2018) - et al.
Structured-light stereo: comparative analysis and integration of structured light and active stereo for measuring dynamic shape
Opt. Lasers Eng.
(2013) - et al.
Metrological evaluation of KinectFusion and its comparison with Microsoft Kinect sensor
Measurement
(2015) - et al.
SR-4000 and CamCube3.0 Time of Flight (ToF) cameras: Tests and Comparison
Remote Sens.
(2012) - et al.
“Review of stereo vision algorithms and their suitability for resource-limited systems
J. Real-Time Image Proc.
(2016) Structured light 3D surface imaging: a tutorial
Adv. Opt. Photonics
(2011)- et al.
High-resolution, real time 3D absolute coordinate measurement based on a phase-shifting method
Opt. Express.
(2006) - et al.
Fourier transform profilometry for the automatic measurement of 3D object shapes
Appl. Opt.
(1983)
Motion-induced error reduction by combining Fourier transform profilometry with phase-shifting profilometry
Opt Express
One-shot 3D scanning by combining sparse landmarks with dense gradient information
Opt. Lasers Eng.
Dynamic scene reconstruction using a single structured light pattern
Dense one-shot 3D reconstruction by detecting continuous regions with parallel line projection
Proc. Int. Conf. Computer Vision
Spacetime stereo: a unifying framework for depth from triangulation
IEEE Trans. Pattern Anal. Mach. Intell.
Cited by (30)
Method for three-dimensional reconstruction of dynamic stereo vision based on line structured light using global optimization
2024, Measurement: Journal of the International Measurement ConfederationA virtual binocular line-structured light measurement method based on a plane mirror
2022, Optics CommunicationsSystematic method for evaluating the performance of three-dimensional optical scanners by structured light projection applied to ballistic vests tests
2022, Measurement: Journal of the International Measurement ConfederationCitation Excerpt :It is worth mentioning that the caliper presents itself as the preferred measuring instrument used in this context, mainly due to the low cost involved, although there are studies [31,32] that strongly suggest the use of three-dimensional digitalization technologies for replacing conventional instruments in various applications. This type of equipment is characterized by having as operating principle the projection of luminous patterns or structures on a certain object and posterior capture of images with the same patterns with some type of distortion [10,33]. Several configurations are possible, using different types of projectors and cameras that capture different images of the same object providing data so that image processing algorithms can identify the same point in different images [34].
Fast adaptive phase unwrapping algorithm based on improved bucket sorting
2021, Optics and Lasers in EngineeringCitation Excerpt :At the same time, it is of great practical significance to study efficient and accurate active structured light measurement technology [1-3]. Active structured light reconstruction technology is based on auxiliary light source projection for 3D reconstruction and measurement, among which the most representative is the phase measurement method [4,5]. The phase measurement method [6,7] mainly uses the multi-step phase shift method to project the grating to the measured object, and combines the phase unwrapping algorithm to obtain the surface shape of the object.
Shape from apparent contours for bent pipes with constant diameter under perspective projection
2021, Measurement: Journal of the International Measurement Confederation