Automatic registration of overlapping 3D point clouds using closest points
Introduction
Traditional TV and modern digital CCD cameras simulate single human eyes, outputting projective images. In contrast, laser scanning systems (range camera, laser range finder), as illustrated in Fig. 1, simulate with a point laser and a digital CCD camera not only both human eyes, but human brains as well. The 3D imaging geometry of laser scanning systems creates a stereo vision and/or performs some function of human brains that post-process some measures of interest and then output range images (Fig. 5, Fig. 7), depicting 3D information of objects. It is likely in the future that laser scanning systems become essential components of intelligent systems due simply to the fact that laser scanning systems directly capture depth information and the recovery of depth information from projective images is often sensitive to noise [50] caused by point sampling on object surface, various illumination conditions, surface orientation, and reflective properties.
However, before this ambition comes true, a number of challenges still have to be tackled. 3D point clouds are described in local camera centred coordinate systems. Due to limited field of view, a number of images have to be captured from different viewpoints so that a full coverage of object surface can be obtained. For the construction of object surface model, these images have to be registered and fused in a single global coordinate system. The automatic registration of overlapping 3D point clouds is one of these challenges and is a fundamental problem for the application of the latest laser scanning techniques in numerous areas, such as object modelling [18], 3D object recognition [23], 3D map construction [20], and simultaneous localization and map building (SLAM) [49]. Overlapping point cloud registration has two goals: one is to determine correspondences between different data sets representing the same free form shape from different viewpoints, the other is to estimate the camera motion parameters bringing one data set into alignment with the other. Fixing either of these two goals renders the other easier. In practice, these two goals are often interwoven, thus complicating overlapping point cloud registration [16], [29].
In the last two decades, a large number of algorithms have been proposed to tackle the difficult and challenging 3D free form shape registration problem based on techniques, such as scatter matrix [25], iterative closest point (ICP) [3], [4], [57], improved ICP algorithm [11], [18], [20], [22], [28], [29], [30], [41], [44], [54], interactive method [52], geometric histogram [1], graduated assignment algorithm [5], [7], [10], [15], [16], [17] among many others. Among these methods, the ideas of the ICP algorithm and the graduated assignment algorithm are most attractive and their brief analysis is thus given below.
While the ICP algorithm assumes that given the initial camera motion parameters rotation matrix R and translation vector t, for any point p in the first image, the closest point p′ in the second to the transformed point Rp+t is its possible correspondent, this idea is so practical and effective that the ICP algorithm has become a de facto standard method for 3D free form shape registration. However, due to occlusion and appearance and disappearance of points in different views and inaccurate initial camera motion parameters, the closest point criterion (CPC) establishes just pseudo-correspondences, since strictly speaking, none of them may represent real correspondences especially at the beginning of iterative registration. However, the relatively accurate correspondences can be selected and used for camera motion re-estimation. With this process iterated, the camera motion parameters can be finally recovered, leading the overlapping point clouds to be accurately aligned. As a result, the key to successfully applying the ICP algorithm for 3D free form shape registration lies in eliminating the correspondences with relatively lower qualities. For this purpose, the following techniques have been proposed:
- •
Increase the dimensionality of points from 3D [3], [4] to higher dimensions by incorporating other geometric or optical features, such as normal vectors [12], invariants [44], curvature [55], laser reflectance strength value [35], and colours [22];
- •
Sample points uniformly [51], in normal space [42], or based on covariance matrix [14];
- •
Establish correspondences from matching points to matching curves [48], [53] to matching 2D images [20], [23], [54], from matching local structural features to examining motion consistency [28], [31], [41] to combining both [27];
- •
Consider the reliability of point correspondences as a function of the cosine of the including angle between vertex normals and their viewing directions [51];
- •
Eliminate false matches through removing boundary points [42], [51], checking interpoint distance consistency [11] or orientation consistency [38], [57], examining motion consistency [29], [30], [31], [41], or examining both the motion and structural consistency [28], [27];
- •
Estimate motion parameters from using least squares to using weighted least squares [51], genetic algorithm [45], M-estimator [35] or simulated annealing [33], or from the Euclidean space to the frequency space [32].
All these different methods have their own advantages and disadvantages and in practice, they can succeed in one situation, but degenerate catastrophically in another. Thus, algorithms that are of general use, accurate, and stable are still desired to be developed for real applications.
Given two overlapping point clouds, the task of registration is to assign a point in one point cloud to a point in another so that these two points represent the same physical point in 3D space. Thus, point cloud registration can be treated as an assignment problem. Intuitively: if point p′ in the second point cloud is the correspondent of point p in the first, then point p should also be the correspondent of point p′. This constraint is called a two-way constraint. However, in practice, once this two-way constraint is enforced, the algorithm often becomes much more complex not only in establishing possible point correspondences, but the optimisation of camera motion parameters as well, given the possible point correspondences between the point clouds being registered.
Similar problem arises in statistical physics: how to decide the equilibrium state of a spin from its possible q states due to external changes such as temperature, pressure, or magnetic fields [56]. This is inherently a combinatorial problem and the exact solution has not yet been found [40]. Instead of insisting on a single state for a spin to select, the mean field theory [56] allows a single spin to select more than one state, each weighted by a probability in the unit interval following the Gibbs distribution. This probability is often called a mean field variable, in contrast with a spin variable which can only take two values, either 0 or 1, indicating whether a particular state is selected. The mean field variable is estimated by the entropy maximization principle [21] and optimised by a stochastic optimisation scheme in the form of deterministic annealing over a new free energy, consisting of a weighted cost minus the entropy of the mean field variables. With the annealing progressing, the free energy converges to a local minimum, leading the mean field variables to approach the spin variables.
The mean field theory provides an approximation to the combinatorial optimisation, where the spin variables are approximated by the mean field variables and the global optimality has not yet been established [40]. However, the mean field theory often provides satisfactory solutions to a large range of problems, such as texture segmentation [39], reinforcement learning [43], data clustering [19], and evidence clustering [2].
The same idea was first employed [15] for graph matching and later [16] for shape matching and the resulting matching algorithm is called the SoftAssign algorithm. Recently, the idea was extended [8] for the alignment of a 3D model and its projective image. Instead of imposing the two-way constraint, the idea was extended through imposing a one-way constraint [10], [17] for shape matching and the resulting algorithm is referred to as the EMICP algorithm. While the two-way constraint requires that the probabilities for a single point in one image to match more than one point in the other must add up to one, and vice versa, the one-way constraint only requires that the probabilities for a single point in one image to match more than one point in another must add up to one. Compared with the applications of the mean field theory in areas like texture segmentation [39], reinforcement learning [43], data clustering [19], or evidence clustering [2], shape matching often possesses a unique characteristic that it has to handle occlusion and appearance and disappearance of points in the data for matching. As a result, slack variables are often introduced to explicitly model them. While the one-way constraint is relatively easy to impose, the two-way constraint can be imposed using the Sinkhorn iterative alternate row and column normalization procedure [46]. The SoftAssign and EMICP algorithms have the following common characteristics:
- •
They both apply the entropy maximization principle to estimate the probabilities of point correspondences that decrease exponentially with respect to the distance between them. This is justified by psychological studies [47] and information theory [21]. Psychological studies show that the probability for a stimulus to ambiguate another decreases exponentially with regard to the distance between these two stimuli. This implies that the points farther away from real correspondences are unlikely to prevent accurate free form shape matching. Information theory shows that the entropy maximization principle leads to the least biased estimate possible on the given information [21]. As a result, the resulting algorithms are expected to produce good registration results;
- •
The mean field variables are optimised using the efficient deterministic annealing scheme which makes sure that the mean field variables eventually approach to an utmost extent the spin variables. Consequently, real point correspondences between the free form shapes being matched can be established;
- •
Both the SoftAssign and EMICP algorithms can be treated as an EM algorithm [9]: in the expectation step, the camera motion parameters are fixed and the correspondence matrix M is then estimated; in the maximization step, the correspondence matrix M is fixed and the camera motion parameters are then estimated in the weighted least squares sense; and
- •
Both the SoftAssign and EMICP algorithms explicitly take into account occlusion and appearance and disappearance of points in the process of registration, leading them to ground on real scenarios.
As a result, both the SoftAssign and EMICP algorithms are theoretically elegant. Practically, once they correctly match overlapping 3D free form shapes with small motions, then they can match the overlapping 3D free form shapes subject to a motion with a rotation angle up to 90° around a certain rotation axis. This characteristic implies that, in practice, the number of images to be captured, and thus time required for processing, can be greatly reduced. This property is critical for applications like real time robot navigation [49].
However, both the SoftAssign and EMICP algorithms have a fatal shortcoming in that their time and space complexities are too high [24] for the registration of overlapping 3D point clouds with thousands of points, which will present obstacles to their applications. In this case, resampling [26], feature point extraction and resampling [5], feature point extraction and fusion [7], image point decimation [17], or feature point extraction [10] has to be used to reduce the number of points for feasible registration. Unfortunately, resampling makes it difficult to replicate the algorithm's performance as different resampling schemes may lead to different results. Since feature extraction is often sensitive to noise due to resampling of points on object surface, quantization of measurement, shape discontinuity, and different optical characteristics of object surface, feature extraction itself is also a challenging problem in the machine vision literatures. Hence, feature extraction and resampling are just an expedient solution to the problem and they do not truly solve the problem, but only transform one difficulty to another.
The mean field theory renders the combinatorial optimisation practical. However, this does not imply that all combinations are necessary for optimisation. In order to address the shortcomings discussed above of the ICP, SoftAssign and EMICP algorithms with an attempt to develop an algorithm that is of general use, efficient, accurate and stable, we wonder in this paper whether it is really necessary to consider for registration all combinations of points in different point clouds, as is the case for both the SoftAssign and EMICP algorithms. Thus, we propose using the CPC established point correspondences in conjunction with either the one-way or two-way constraint for registration with the probability of each point correspondence estimated by the mean field annealing scheme. For the sake of computational efficiency, the optimised k-D tree [13] is employed for the search of the closest points. Due to the nearly ubiquitous occurrence of occlusion and appearance and disappearance of points in different views, slack variables have to be introduced to explicitly model outliers in the process of registration. The motion parameters of interest are finally estimated by the quaternion method [3] in the weighted least squares sense. Thus, the improved ICP algorithm combines the advantages of accuracy of both the SoftAssign and EMICP algorithms and efficiency of the traditional ICP algorithm and it only assumes that the 3D point clouds to be registered are represented as either sparse or dense unorganised points. This assumption is reasonable since other representations of 3D free form shapes like line segments, triangular meshes, planar patches, or analytic forms can all be transformed into points [3].
While the traditional ICP algorithm and most of its variants require to classify possible point matches established into either real or false ones [12], [29], [31], [38], [41], [51], [57], they often require either to extract structural (geometric or optical) features from image data or to estimate some motion parameters of interest from the data corrupted by outliers. In these cases, the classification is unlikely always successful due to the fact that the point matches established by the traditional CPC are of comparable quality at each iteration of the algorithm. It is unlikely that some point matches are of high quality, while their neighbours are of significantly lower quality. In contrast, the proposed ICP variant does not involve any additional processing of either point clouds being registered or possible correspondences established, uniquely treats all the possible point matches in the sense of estimating their probability as being real ones, and thus avoids the classification difficulty and has an advantage of easy implementation.
For a comparative study of performance, we also implement the Geometric ICP algorithm (GICP) [31] based on rigid motion constraints. The main reason why the GICP algorithm was chosen is that while most existing algorithms require more or less structural information from images [11], [20], [22], [28], [38], [44], [54], such information is not easy to estimate from point clouds. The GICP algorithm was designed for the registration of unorganised point sets and does not need to extract any structural information from point clouds and thus, also has an advantage of easy implementation. While the improved ICP algorithm applies either the two-way or one-way constraint for the evaluation of possible point correspondences established by the traditional CPC, the GICP algorithm applies the rigid motion constraints to reject false matches. Such a comparative study is valuable since it can reveal which strategy is more effective for the evaluation of possible point matches established by the traditional ICP criterion.
The rest of this paper is structured as follows: Section 2 outlines the SoftAssign algorithm and proposes its extension, Section 3 outlines the EMICP algorithm and also proposes its extension, while Section 4 presents experimental results. Finally, Section 5 discusses a number of issues in the process of registration and draws some conclusions.
Section snippets
The outline of the SoftAssign algorithm and its extension
The following notations are used throughout this paper: capital letters denote vectors or matrices, lower case letters denote scalars, ‖·‖ denotes the Euclidean norm of a vector, parameters with and without ∧ in Section 4 denote calibrated and real ones.
Assume that the two point clouds to be registered are represented as two sets of unorganised points P={p1, p2,…,pn1} and , representing the same free form shape from two different nearby viewpoints with overlap in 3D space. Due
Outline of the EMICP algorithm
The following computational steps are proposed in [10] for free form point pattern matching:
- •
Compute the probability αij that pi matches , , and ; and
- •
Construct a new objective function for the free form point pattern matching
The
Experimental results
In order to provide a better understanding of performance of the proposed SoftICP and WeightICP algorithms, we also implemented the original SoftAssign and EMICP algorithms and the geometric ICP (GICP) algorithm [31] for a comparative study based on both synthetic data and real images. All algorithms were directly applied to the synthetic data and real images without any pre-processing, feature extraction, or image segmentation and also without any knowledge about the distribution of points,
Discussion
Through the experiments presented above based on both synthetic data and real images, the following observation has been made:
- •
The reason why the proposed SoftICP and WeightICP algorithms work can be explained from five aspects: high quality point matches established, probability estimation from the entropy maximization principle and explicit outlier modelling, probability optimization using the deterministic annealing scheme, the nature of camera motion estimation and point match
Acknowledgements
We would like to express our sincere thanks to the two anonymous reviewers for their valuable comments that have improved the readability of the paper.
References (57)
- et al.
Object modelling by registration of multiple range images
IVC
(1992) - et al.
A new point matching algorithm for non-rigid registration
CVIU
(2003) - et al.
A unified non-rigid feature registration method for brain mapping
Med. Image Anal.
(2003) - et al.
3D–2D projective registration of free-form curves and surfaces
CVIU
(1997) - et al.
New algorithms for 2-D and 3-D point matching: pose estimation and correspondence
Pattern Recognit.
(1998) - et al.
Fully automatic registration of multiple 3D data sets
IVC
(2003) - et al.
Registration and integration of textured 3-D data
IVC
(1999) - et al.
Developing structural constraints for accurate registration of overlapping range images
Rob. Autonomous Syst.
(2004) Improving ICP with easy implementation for free form surface matching
Pattern Recognit.
(2004)- et al.
Multiscale annealing for grouping and unsupervised texture segmentation
CVIU
(1999)
On the representation of rigid body transformations for accurate registration of free form shapes
Rob. Autonomous Syst.
An autonomous mobile robot with a 3D laser range finder for 3D exploration and digitalisation of indoor environment
Rob. Autonomous Syst.
Dempster-shafer clustering using potts spin mean field theory
Soft Comput.
A method for registration of 3D shapes
IEEE Trans. PAMI
Linear Programming
Maximum likelihood from incomplete data via the EM algorithm
J. R. Stat. Soc. Ser. B
Hand motion from 3D point trajectories and a smooth surface model
ECCV
An algorithm for finding best matches in logarithmic expected time
ACM Trans. Math. Software
A graduated assignment algorithm for graph matching
IEEE PAMI
Pairwise data clustering by deterministic annealing
IEEE Trans. PAMI
Information theory and statistical mechanics
Phys. Rev.
Using spin images for efficient object recognition in cluttered 3D scenes
IEEE Trans. PAMI
Cited by (83)
An integrated approach of reverse engineering aided remanufacturing process for worn components
2017, Robotics and Computer-Integrated ManufacturingCitation Excerpt :Another group strived to formulate an optimization transformation to achieve a best-fit positon between two models through an iterative procedure, such as genetic algorithm [17], motion approximation [18], simulated annealing [19,20], etc., which is capable of exploring the search space of candidate solutions. Among those iteration methods, iterative closest point (ICP) algorithm [21] and its variants [22–26] are dominant techniques and widely used for robust rigid registration of 3D data. When ICP is used to match models with partial overlap and unavoidably introduces false point matches, various techniques have been developed to improve the registration results.
Efficient rotation estimation for 3D registration and global localization in structured point clouds
2017, Image and Vision ComputingCitation Excerpt :Many existing 3D registration algorithms consider the rotation and translation estimation task as an iterative optimization problem. The ICP algorithm [15] and its variants [16,25,27-33] have been frequently used to perform 3D point cloud registration. ICP is one of the earliest point cloud registration techniques, it uses a least square method to achieve optimal matching between two point clouds.
Research progress of point cloud registration technology based on industrial 3D inspection
2023, Jingangshi yu Moliao Moju Gongcheng/Diamond and Abrasives EngineeringPoint cloud registration method for deformed thin-walled parts based on on-machine measurement of structured light
2022, Guangdianzi Jiguang/Journal of Optoelectronics Laser