Fig. 1. The hand model as polygonal surface (top). The hand model and its degrees of freedom (bottom), where DIP indicates the ‘distal interphalangeal’ joints, PIP the ‘proximal interphalangeal’ joints, MP the ‘metacarpophalangeal’ joints, IP the ‘interphalangeal’ and TM the ‘trapeziometacarpal’ joints.

Fig. 2. Input image of the ShapeSnatcher (a). Depth map extracted from the scene (a) and masked out by skin color segmentation (b).

Fig. 3. The calculation of the error function. An approximation to the closest point on the surface is calculated by projecting the model point onto the tangential plane at . This yields the approximation x_c. is the depth map provided by the 3D sensor in camera coordinates. Furthermore, the difference between the observed normal n and the predicted normal is taken into account of the cost function.

Fig. 4. Sketch of the assumed displacement of x_c given a displacement of .

Fig. 5. On the left (a), evolution of the tracker's speed according to the number of sample points and the sampling mode. On the right (b), illustration of a random subsampling performed on the hand model with 2 points per phalanx and 15 points for the visible part of the palm. When a hand part is not visible, the points expected on it are randomly chosen on any other visible part.

Fig. 6. Original sequence and 3D reconstructions of the experience presented in Fig. 5(a).

Fig. 7. Comparison of the accumulated number of iterations when tracking with or without inter-frame step size adaptation. The solid line is with inter-frame initialization, the dashed line when step sizes are simply reset at the beginning of each frame.

Fig. 8. Evolution of the SMD algorithm while tracking. The top row shows this evolution projected on the translation in X and rotation in Z of the palm and the bottom rows present the resulting sequence.

Fig. 9. Evolution of the gradient descent algorithm ((a) and (b)) and a deterministic sampling ((c) and (d)) while tracking the same sequence as in Fig. 8. (b) Is the final result of the gradient descent after 16 iterations: the hand model is slightly shifted from the target. (d) Is the final result of the deterministic sampling after nine iterations: this approach clearly diverges from the correct solution.

Fig. 10. Comparison between SMD and alternative optimization algorithms. From top to bottom: the results of the SMD using a cost function E evaluating the distances and the differences in surface orientations, SMD using a cost function E only evaluating the distances, gradient descent, Powell and annealed particle filter (APF). The model is visualized as the vertices of the skin polygonal representation. The pictures have been cropped for better visibility, but see Fig. 11 for examples of similar, complete frames.

Fig. 11. Hand tracking with self-occlusions. Far right: 3D model for frame 103 (top view).

Fig. 12. Spelling the word ‘FLY’ in American sign language with SMD: top row represents the ‘F’ letter, the second row the ‘L’ letter and the last row the ‘Y’ letter. The last column is the 3D reconstruction from the third one.

Comparison of the computation time of the Fig. 10