doi:10.1016/S0031-3203(99)00059-X 
Copyright © 2000 Pattern Recognition Society. Published by Elsevier Science B.V.
Localizing a polyhedral object in a robot hand by integrating visual and tactile data*1
Department of Computing Science, University of Alberta, Edmonton, Alberta, Canada T6G 2H1
Received 28 September 1998;
revised 2 March 1999;
accepted 2 March 1999.
Available online 9 December 1999.
References and further reading may be available for this article. To view references and further reading you must
purchase this article.
Abstract
We present a novel technique for localizing a polyhedral object in a robot hand by integrating visual and tactile data. Localization is performed by matching a hybrid set of visual and tactile features with corresponding model features. The matching process first determines a subset of the object's six degrees of freedom (DOFs) using the tactile feature. The remaining DOFs, which cannot be determined from the tactile feature, are then obtained by matching the visual feature. A couple of touch and vision/touch-based filtering techniques are developed to reduce the number of model feature sets that are actually matched with a given scene set. We demonstrate the performance of the technique using simulated and real data. In particular, we show its superiority over vision-based localization in the following aspects: (1) capability of determining the object pose under heavy occlusion, (2) number of generated pose hypotheses, and (3) accuracy of estimating the object depth.
Author Keywords: 3D object recognition; Pose estimation; Visual data; Tactile data; Sensor integration; Robot hand; Object manipulation
Fig. 1. Cases of tactile contact and associated tactile-sensor frames: (a) surface contact (S-patch): (1) origin: center of the patch, (2) z-axis: outward normal of the model surface in contact with the patch, (3) x- and y-axes: placed arbitrarily in the tactile-patch plane, (b) surface-edge contact (SE-patch): (1) origin: mid point of the edge, (2) z-axis: outward normal of the model surface in contact with the patch, (3) y-axis: normal to the edge toward the interior of the patch, (4) x-axis: along the edge such that the right-hand rule is satisfied.
Fig. 2. An illustration of the matching process using visual and tactile features: (a) S-patch case, (b) SE-patch case. Note that c is the camera viewpoint.
Fig. 3. Transformation of S-surface Si to the tactile S-patch: (a) after rotation about the x- and y-axes, (b) after translation along the z-axis.
Fig. 4. Transformation of model SE-surface (Si, Ej) to the tactile SE-patch: (a) after rotation about the z-axis, (b) after translation along the y-axis.
Fig. 5. An illustration of the matching process: (a) partially transformed model junction iJm, (b) visual junction Js, (c) a match between iJm and Js to form fJm.
Fig. 6. Matching in the S-patch case: (a) computing rotation, (b) computing translation.
Fig. 7. Matching in the SE-patch case.
Fig. 8. Application of a volumetric constraint, iV: (a) iV is inconsistent with junction Js, (b) iV is consistent with junction Js.
Fig. 9. Volumetric constraints: (a) S-patch case, (b) SE-patch case.
Fig. 10. Representation of vectors that lie inside half plane Psk.
Fig. 11. Computing bounds on θ: Bounds [αmin, αmax] and [αmink, αmaxk] correspond to the sections of L(vs) and L(usk) that lie inside volume iV. In such a case, we have θ
[θmin, θmax].
Fig. 12. A synthetic scene of a peg being inserted to a slot: (a) scene, (b) extracted edges.
Fig. 13. Peg-to-a-slot scene: Positions of the tactile sensor: (a) surface contact, (b) surface-edge contact.
Fig. 14. First real scene: (a) scene, (b) extracted edges.
Fig. 15. First real scene: (a) original tactile image, (b) thresholded image.
Fig. 16. First real scene: Valid hypotheses for: (a) vision-only case (six hypotheses), (b) S-patch case (four hypotheses), (c) SE-patch case (four hypotheses).
Fig .17. Second real scene: (a) scene, (b) extracted edges.
Fig. 18. Second real scene: (a) original tactile image, (b) thresholded image.
Fig. 19. Second real scene: valid hypothesis for: (a) S-patch case, (b) SE-patch case.
Table 1. A comparison between visual and tactile data
Table 2. Peg-to-a-slot scene: generated hypotheses
Table 3. Peg-to-a-slot scene: Errors in estimating the object pose
Table 4. First real scene: generated hypotheses
Table 5. Second real scene: generated hypotheses (all visually generated hypotheses are erroneous)
Abstract
Purchase PDF (713 K)
E-mail Article
Add to my Quick Links
Cited By in Scopus (0)
Purchase PDF (616 K)
