Automatic registration of overlapping 3D point clouds using closest points

https://doi.org/10.1016/j.imavis.2006.01.009Get rights and content

Abstract

While the SoftAssign algorithm imposes a two-way constraint embedded into the deterministic annealing scheme and the EMICP algorithm imposes a one-way constraint, they represent the state of the art technique for the automatic registration of overlapping free form shapes. They both have a time complexity of O(n2). While the former has a space complexity also of O(n2), the latter has a space complexity of O(n). The heavy demand for computation and storage memory renders either the SoftAssign or EMICP algorithm to hardly operate on whole shapes with thousands of points. In this case, they often have to reduce the number of points to an order of 100s on the free form shapes to be registered. This paper proposes using closest points in conjunction with either the one-way or two-way constraint for the automatic registration of overlapping 3D point clouds and thus, combining the accuracy of both the SoftAssign and EMICP algorithms with the efficiency of the traditional ICP algorithm. A comparative study based on both synthetic data and real images has shown that the proposed algorithm does not significantly sacrifice accuracy and stability of either the SoftAssign or EMICP algorithm, but gains remarkable efficiency of the traditional ICP algorithm for the automatic registration of overlapping 3D point clouds. Since, the proposed algorithm is of general use and has an advantage of easy implementation, it is likely to become in the future a benchmark for the automatic registration of overlapping 3D point clouds.

Introduction

Traditional TV and modern digital CCD cameras simulate single human eyes, outputting projective images. In contrast, laser scanning systems (range camera, laser range finder), as illustrated in Fig. 1, simulate with a point laser and a digital CCD camera not only both human eyes, but human brains as well. The 3D imaging geometry of laser scanning systems creates a stereo vision and/or performs some function of human brains that post-process some measures of interest and then output range images (Fig. 5, Fig. 7), depicting 3D information of objects. It is likely in the future that laser scanning systems become essential components of intelligent systems due simply to the fact that laser scanning systems directly capture depth information and the recovery of depth information from projective images is often sensitive to noise [50] caused by point sampling on object surface, various illumination conditions, surface orientation, and reflective properties.

However, before this ambition comes true, a number of challenges still have to be tackled. 3D point clouds are described in local camera centred coordinate systems. Due to limited field of view, a number of images have to be captured from different viewpoints so that a full coverage of object surface can be obtained. For the construction of object surface model, these images have to be registered and fused in a single global coordinate system. The automatic registration of overlapping 3D point clouds is one of these challenges and is a fundamental problem for the application of the latest laser scanning techniques in numerous areas, such as object modelling [18], 3D object recognition [23], 3D map construction [20], and simultaneous localization and map building (SLAM) [49]. Overlapping point cloud registration has two goals: one is to determine correspondences between different data sets representing the same free form shape from different viewpoints, the other is to estimate the camera motion parameters bringing one data set into alignment with the other. Fixing either of these two goals renders the other easier. In practice, these two goals are often interwoven, thus complicating overlapping point cloud registration [16], [29].

In the last two decades, a large number of algorithms have been proposed to tackle the difficult and challenging 3D free form shape registration problem based on techniques, such as scatter matrix [25], iterative closest point (ICP) [3], [4], [57], improved ICP algorithm [11], [18], [20], [22], [28], [29], [30], [41], [44], [54], interactive method [52], geometric histogram [1], graduated assignment algorithm [5], [7], [10], [15], [16], [17] among many others. Among these methods, the ideas of the ICP algorithm and the graduated assignment algorithm are most attractive and their brief analysis is thus given below.

While the ICP algorithm assumes that given the initial camera motion parameters rotation matrix R and translation vector t, for any point p in the first image, the closest point p′ in the second to the transformed point Rp+t is its possible correspondent, this idea is so practical and effective that the ICP algorithm has become a de facto standard method for 3D free form shape registration. However, due to occlusion and appearance and disappearance of points in different views and inaccurate initial camera motion parameters, the closest point criterion (CPC) establishes just pseudo-correspondences, since strictly speaking, none of them may represent real correspondences especially at the beginning of iterative registration. However, the relatively accurate correspondences can be selected and used for camera motion re-estimation. With this process iterated, the camera motion parameters can be finally recovered, leading the overlapping point clouds to be accurately aligned. As a result, the key to successfully applying the ICP algorithm for 3D free form shape registration lies in eliminating the correspondences with relatively lower qualities. For this purpose, the following techniques have been proposed:

  • Increase the dimensionality of points from 3D [3], [4] to higher dimensions by incorporating other geometric or optical features, such as normal vectors [12], invariants [44], curvature [55], laser reflectance strength value [35], and colours [22];

  • Sample points uniformly [51], in normal space [42], or based on covariance matrix [14];

  • Establish correspondences from matching points to matching curves [48], [53] to matching 2D images [20], [23], [54], from matching local structural features to examining motion consistency [28], [31], [41] to combining both [27];

  • Consider the reliability of point correspondences as a function of the cosine of the including angle between vertex normals and their viewing directions [51];

  • Eliminate false matches through removing boundary points [42], [51], checking interpoint distance consistency [11] or orientation consistency [38], [57], examining motion consistency [29], [30], [31], [41], or examining both the motion and structural consistency [28], [27];

  • Estimate motion parameters from using least squares to using weighted least squares [51], genetic algorithm [45], M-estimator [35] or simulated annealing [33], or from the Euclidean space to the frequency space [32].

All these different methods have their own advantages and disadvantages and in practice, they can succeed in one situation, but degenerate catastrophically in another. Thus, algorithms that are of general use, accurate, and stable are still desired to be developed for real applications.

Given two overlapping point clouds, the task of registration is to assign a point in one point cloud to a point in another so that these two points represent the same physical point in 3D space. Thus, point cloud registration can be treated as an assignment problem. Intuitively: if point p′ in the second point cloud is the correspondent of point p in the first, then point p should also be the correspondent of point p′. This constraint is called a two-way constraint. However, in practice, once this two-way constraint is enforced, the algorithm often becomes much more complex not only in establishing possible point correspondences, but the optimisation of camera motion parameters as well, given the possible point correspondences between the point clouds being registered.

Similar problem arises in statistical physics: how to decide the equilibrium state of a spin from its possible q states due to external changes such as temperature, pressure, or magnetic fields [56]. This is inherently a combinatorial problem and the exact solution has not yet been found [40]. Instead of insisting on a single state for a spin to select, the mean field theory [56] allows a single spin to select more than one state, each weighted by a probability in the unit interval following the Gibbs distribution. This probability is often called a mean field variable, in contrast with a spin variable which can only take two values, either 0 or 1, indicating whether a particular state is selected. The mean field variable is estimated by the entropy maximization principle [21] and optimised by a stochastic optimisation scheme in the form of deterministic annealing over a new free energy, consisting of a weighted cost minus the entropy of the mean field variables. With the annealing progressing, the free energy converges to a local minimum, leading the mean field variables to approach the spin variables.

The mean field theory provides an approximation to the combinatorial optimisation, where the spin variables are approximated by the mean field variables and the global optimality has not yet been established [40]. However, the mean field theory often provides satisfactory solutions to a large range of problems, such as texture segmentation [39], reinforcement learning [43], data clustering [19], and evidence clustering [2].

The same idea was first employed [15] for graph matching and later [16] for shape matching and the resulting matching algorithm is called the SoftAssign algorithm. Recently, the idea was extended [8] for the alignment of a 3D model and its projective image. Instead of imposing the two-way constraint, the idea was extended through imposing a one-way constraint [10], [17] for shape matching and the resulting algorithm is referred to as the EMICP algorithm. While the two-way constraint requires that the probabilities for a single point in one image to match more than one point in the other must add up to one, and vice versa, the one-way constraint only requires that the probabilities for a single point in one image to match more than one point in another must add up to one. Compared with the applications of the mean field theory in areas like texture segmentation [39], reinforcement learning [43], data clustering [19], or evidence clustering [2], shape matching often possesses a unique characteristic that it has to handle occlusion and appearance and disappearance of points in the data for matching. As a result, slack variables are often introduced to explicitly model them. While the one-way constraint is relatively easy to impose, the two-way constraint can be imposed using the Sinkhorn iterative alternate row and column normalization procedure [46]. The SoftAssign and EMICP algorithms have the following common characteristics:

  • They both apply the entropy maximization principle to estimate the probabilities of point correspondences that decrease exponentially with respect to the distance between them. This is justified by psychological studies [47] and information theory [21]. Psychological studies show that the probability for a stimulus to ambiguate another decreases exponentially with regard to the distance between these two stimuli. This implies that the points farther away from real correspondences are unlikely to prevent accurate free form shape matching. Information theory shows that the entropy maximization principle leads to the least biased estimate possible on the given information [21]. As a result, the resulting algorithms are expected to produce good registration results;

  • The mean field variables are optimised using the efficient deterministic annealing scheme which makes sure that the mean field variables eventually approach to an utmost extent the spin variables. Consequently, real point correspondences between the free form shapes being matched can be established;

  • Both the SoftAssign and EMICP algorithms can be treated as an EM algorithm [9]: in the expectation step, the camera motion parameters are fixed and the correspondence matrix M is then estimated; in the maximization step, the correspondence matrix M is fixed and the camera motion parameters are then estimated in the weighted least squares sense; and

  • Both the SoftAssign and EMICP algorithms explicitly take into account occlusion and appearance and disappearance of points in the process of registration, leading them to ground on real scenarios.

As a result, both the SoftAssign and EMICP algorithms are theoretically elegant. Practically, once they correctly match overlapping 3D free form shapes with small motions, then they can match the overlapping 3D free form shapes subject to a motion with a rotation angle up to 90° around a certain rotation axis. This characteristic implies that, in practice, the number of images to be captured, and thus time required for processing, can be greatly reduced. This property is critical for applications like real time robot navigation [49].

However, both the SoftAssign and EMICP algorithms have a fatal shortcoming in that their time and space complexities are too high [24] for the registration of overlapping 3D point clouds with thousands of points, which will present obstacles to their applications. In this case, resampling [26], feature point extraction and resampling [5], feature point extraction and fusion [7], image point decimation [17], or feature point extraction [10] has to be used to reduce the number of points for feasible registration. Unfortunately, resampling makes it difficult to replicate the algorithm's performance as different resampling schemes may lead to different results. Since feature extraction is often sensitive to noise due to resampling of points on object surface, quantization of measurement, shape discontinuity, and different optical characteristics of object surface, feature extraction itself is also a challenging problem in the machine vision literatures. Hence, feature extraction and resampling are just an expedient solution to the problem and they do not truly solve the problem, but only transform one difficulty to another.

The mean field theory renders the combinatorial optimisation practical. However, this does not imply that all combinations are necessary for optimisation. In order to address the shortcomings discussed above of the ICP, SoftAssign and EMICP algorithms with an attempt to develop an algorithm that is of general use, efficient, accurate and stable, we wonder in this paper whether it is really necessary to consider for registration all combinations of points in different point clouds, as is the case for both the SoftAssign and EMICP algorithms. Thus, we propose using the CPC established point correspondences in conjunction with either the one-way or two-way constraint for registration with the probability of each point correspondence estimated by the mean field annealing scheme. For the sake of computational efficiency, the optimised k-D tree [13] is employed for the search of the closest points. Due to the nearly ubiquitous occurrence of occlusion and appearance and disappearance of points in different views, slack variables have to be introduced to explicitly model outliers in the process of registration. The motion parameters of interest are finally estimated by the quaternion method [3] in the weighted least squares sense. Thus, the improved ICP algorithm combines the advantages of accuracy of both the SoftAssign and EMICP algorithms and efficiency of the traditional ICP algorithm and it only assumes that the 3D point clouds to be registered are represented as either sparse or dense unorganised points. This assumption is reasonable since other representations of 3D free form shapes like line segments, triangular meshes, planar patches, or analytic forms can all be transformed into points [3].

While the traditional ICP algorithm and most of its variants require to classify possible point matches established into either real or false ones [12], [29], [31], [38], [41], [51], [57], they often require either to extract structural (geometric or optical) features from image data or to estimate some motion parameters of interest from the data corrupted by outliers. In these cases, the classification is unlikely always successful due to the fact that the point matches established by the traditional CPC are of comparable quality at each iteration of the algorithm. It is unlikely that some point matches are of high quality, while their neighbours are of significantly lower quality. In contrast, the proposed ICP variant does not involve any additional processing of either point clouds being registered or possible correspondences established, uniquely treats all the possible point matches in the sense of estimating their probability as being real ones, and thus avoids the classification difficulty and has an advantage of easy implementation.

For a comparative study of performance, we also implement the Geometric ICP algorithm (GICP) [31] based on rigid motion constraints. The main reason why the GICP algorithm was chosen is that while most existing algorithms require more or less structural information from images [11], [20], [22], [28], [38], [44], [54], such information is not easy to estimate from point clouds. The GICP algorithm was designed for the registration of unorganised point sets and does not need to extract any structural information from point clouds and thus, also has an advantage of easy implementation. While the improved ICP algorithm applies either the two-way or one-way constraint for the evaluation of possible point correspondences established by the traditional CPC, the GICP algorithm applies the rigid motion constraints to reject false matches. Such a comparative study is valuable since it can reveal which strategy is more effective for the evaluation of possible point matches established by the traditional ICP criterion.

The rest of this paper is structured as follows: Section 2 outlines the SoftAssign algorithm and proposes its extension, Section 3 outlines the EMICP algorithm and also proposes its extension, while Section 4 presents experimental results. Finally, Section 5 discusses a number of issues in the process of registration and draws some conclusions.

Section snippets

The outline of the SoftAssign algorithm and its extension

The following notations are used throughout this paper: capital letters denote vectors or matrices, lower case letters denote scalars, ‖·‖ denotes the Euclidean norm of a vector, parameters with and without ∧ in Section 4 denote calibrated and real ones.

Assume that the two point clouds to be registered are represented as two sets of unorganised points P={p1, p2,…,pn1} and P={p1,p2,,pn2}, representing the same free form shape from two different nearby viewpoints with overlap in 3D space. Due

Outline of the EMICP algorithm

The following computational steps are proposed in [10] for free form point pattern matching:

  • Compute the probability αij that pi matches pj:αij=(1/Ci)exp(dij2/σp2), Ci=exp(d02/σp2)+k=1n2exp(dik2/σp2), and dij=||pjRpit||; and

  • Construct a new objective function for the free form point pattern matching

Ep(R,t)=minR,ti=1n1λi2qiRpit2where λi=j=1n2αij and qi=(1/λi)j=1n2αijpj. This objective function can be optimised by the quaternion method [3] in the weighted least squares sense.

The

Experimental results

In order to provide a better understanding of performance of the proposed SoftICP and WeightICP algorithms, we also implemented the original SoftAssign and EMICP algorithms and the geometric ICP (GICP) algorithm [31] for a comparative study based on both synthetic data and real images. All algorithms were directly applied to the synthetic data and real images without any pre-processing, feature extraction, or image segmentation and also without any knowledge about the distribution of points,

Discussion

Through the experiments presented above based on both synthetic data and real images, the following observation has been made:

  • The reason why the proposed SoftICP and WeightICP algorithms work can be explained from five aspects: high quality point matches established, probability estimation from the entropy maximization principle and explicit outlier modelling, probability optimization using the deterministic annealing scheme, the nature of camera motion estimation and point match

Acknowledgements

We would like to express our sincere thanks to the two anonymous reviewers for their valuable comments that have improved the readability of the paper.

References (57)

  • M.A. Rodrigues et al.

    On the representation of rigid body transformations for accurate registration of free form shapes

    Rob. Autonomous Syst.

    (2002)
  • H. Surmann et al.

    An autonomous mobile robot with a 3D laser range finder for 3D exploration and digitalisation of indoor environment

    Rob. Autonomous Syst.

    (2003)
  • A.P. Ashbrook, R.B. Fisher, et al., Finding surface correspondences for object recognition and registration using...
  • M. Bengtsson et al.

    Dempster-shafer clustering using potts spin mean field theory

    Soft Comput.

    (2001)
  • P.J. Besl et al.

    A method for registration of 3D shapes

    IEEE Trans. PAMI

    (1992)
  • V. Chvatal

    Linear Programming

    (1983)
  • P. David, D. DeMenthon, R. Duraiswami, H. Samet. Simultaneous pose and correspondence determination using line...
  • A.P. Dempster et al.

    Maximum likelihood from incomplete data via the EM algorithm

    J. R. Stat. Soc. Ser. B

    (1977)
  • G. Dewaele et al.

    Hand motion from 3D point trajectories and a smooth surface model

    ECCV

    (2004)
  • C. Dorai, G. Wang, A.K. Jain, C. Mercer. From images to models: automatic model construction from multiple views,...
  • J.H. Friedman et al.

    An algorithm for finding best matches in logarithmic expected time

    ACM Trans. Math. Software

    (1977)
  • N. Gelfand, L. Ikemoto, S. Rusinhiewicz, M. Levoy, Geometrically stable sampling for the ICP algorithm, Proceedings of...
  • S. Gold et al.

    A graduated assignment algorithm for graph matching

    IEEE PAMI

    (1996)
  • S. Granger, Xavier Pennec, Multi-scale EMICP: a fact and robust approach for surface registration, Proceedings of the...
  • D. Hahnel, S. Thrun, W. Burgard, An extension of the ICP algorithm for modelling nonrigid objects with mobile robots,...
  • T. Hofmann et al.

    Pairwise data clustering by deterministic annealing

    IEEE Trans. PAMI

    (1997)
  • E.T. Jaynes

    Information theory and statistical mechanics

    Phys. Rev.

    (1957)
  • A. Johnson et al.

    Using spin images for efficient object recognition in cluttered 3D scenes

    IEEE Trans. PAMI

    (1999)
  • Cited by (83)

    • An integrated approach of reverse engineering aided remanufacturing process for worn components

      2017, Robotics and Computer-Integrated Manufacturing
      Citation Excerpt :

      Another group strived to formulate an optimization transformation to achieve a best-fit positon between two models through an iterative procedure, such as genetic algorithm [17], motion approximation [18], simulated annealing [19,20], etc., which is capable of exploring the search space of candidate solutions. Among those iteration methods, iterative closest point (ICP) algorithm [21] and its variants [22–26] are dominant techniques and widely used for robust rigid registration of 3D data. When ICP is used to match models with partial overlap and unavoidably introduces false point matches, various techniques have been developed to improve the registration results.

    • Efficient rotation estimation for 3D registration and global localization in structured point clouds

      2017, Image and Vision Computing
      Citation Excerpt :

      Many existing 3D registration algorithms consider the rotation and translation estimation task as an iterative optimization problem. The ICP algorithm [15] and its variants [16,25,27-33] have been frequently used to perform 3D point cloud registration. ICP is one of the earliest point cloud registration techniques, it uses a least square method to achieve optimal matching between two point clouds.

    • Research progress of point cloud registration technology based on industrial 3D inspection

      2023, Jingangshi yu Moliao Moju Gongcheng/Diamond and Abrasives Engineering
    View all citing articles on Scopus
    View full text