- Global hand pose: using thumb-tip position term (instead of typical wrist position term) and wrist orientation term (with small weight).
- Overall hand shape: using fingertip positions relative to the wrist to preserve the posture of the human hand.
- Relative fingertip positions: using spatial relationship between thumb-tip and primary fingertips to enable precise coordination of fingertips for manipulation tasks like pinching.
- Fingertip orientations: using vectors from DIP joints to fingertips to ensure correct contact normals for fine manipulation.
- Complete retargeting objective: combining all the above terms with joint regularization (prevent undesired joint configurations) and velocity regularization (encourage smooth motion).
Method
Crucial objectives in human-to-robot retargeting for dexterous manipulation. We explore the appropriate formulation of these objectives and experimentally analyze their significance for different manipulation tasks.
Objective Terms
We intuitively list the factors that are potentially crucial to human-to-robot retargeting for dexterous manipulation, formulate their corresponding mathematical representations, and construct a complete retargeting objective. (See full formulations and their explanations in Section II of the paper.)
$$\begin{aligned}
\mathcal{L}_{\text{hand pose}}
&= \left\lVert \mathbf{p}^{\,h}_{\text{thumb}} - \mathbf{p}^{\,r}_{\text{thumb}} \right\rVert_2^2 \\
&\quad + \beta_{\text{rot}}\,
\mathrm{angle}\!\left(\mathbf{q}^{\,h}_{\text{wrist}},\, \mathbf{q}^{\,r}_{\text{wrist}}\right)
\end{aligned}\tag{1}$$
$$\mathcal{L}_{\text{fingertip pos}}
= \sum_{i=1}^{N} \left\lVert \mathbf{v}^{\,h}_{i} - \mathbf{v}^{\,r}_{i} \right\rVert_2^2\tag{2}$$
$$\mathcal{L}_{\text{pinch}}
= \sum_{i=1}^{N-1} s(d_i)\,
\left\lVert \boldsymbol{\gamma}^{\,r}_{i} - l(d_i)\,\hat{\boldsymbol{\gamma}}^{\,h}_{i} \right\rVert_2^2\tag{3}$$
$$\mathcal{L}_{\text{fingertip rot}}
= \sum_{i=1}^{N} \left\lVert \mathbf{r}^{\,h}_{i} - \mathbf{r}^{\,r}_{i} \right\rVert_2^2\tag{5}$$
$$\begin{aligned}
\mathcal{L}_{\text{total}}
&= \lambda_1 \mathcal{L}_{\text{thumb pos}}
+ \lambda_2 \mathcal{L}_{\text{wrist rot}} \\
&\quad + \lambda_3 \mathcal{L}_{\text{fingertip pos}}
+ \lambda_4 \mathcal{L}_{\text{fingertip rot}} \\
&\quad + \lambda_5 \mathcal{L}_{\text{pinch}}
+ \mathcal{L}_{\text{joint}}
+ \mathcal{L}_{\text{vel}}
\end{aligned}\tag{6}$$
Experiments & Results
We analyze the significance of different retargeting objectives via kinematic posture retargeting and real-world teleoperation manipulation.
Ablation Setup
We implement ablations by removing or changing certain objective terms in the above complete objective to analyze their significance.
Kinematic Posture Retargeting
Setup. We evaluate the full retargeting objective and eight ablations across two pieces of offline data: (1) Traj 1: a trajectory of three pinch motions involving the thumb and the index, middle, and ring fingers, respectively; (2) Traj 2: a trajectory involving finger crossing motions. The evaluation is performed on 2 types of robot hands: the LEAP hand and the Shadow hand. We use four quantitative metrics for kinematic postures: 1) average fingertip position error in the global frame, 2) average fingertip position error relative to the wrist, 3) average fingertip position error relative to the thumb, and 4) average fingertip orientation error.
Results of kinematic posture retargeting on Traj 1 (LEAP hand). Each bar shows the error of one ablation implementation and the colors represent the ablation category defined in Table I. For the metrics of fingertip global position and fingertip relative position to the thumb, only the errors of the two fingers involved in the pinching motion are considered.
Results of kinematic posture retargeting on Traj 2 (LEAP hand).
Results of kinematic posture retargeting on Traj 1 (Shadow hand).
Results of kinematic posture retargeting on Traj 2 (Shadow hand).
Real-world kinematic postures retargeting results. We choose 3 ablations (A1, A3, and A7). From left to right, we show the human-hand demonstration, the posture retargeting result using the full objective, and the result under the corresponding ablation.
Human demo
Full objective
Ablation
Ablation 1 (i.e., w/o pinch term): less precise pinch (cannot close the gap between thumb and primary fingertips).
Ablation 3 (i.e., w/o fingertip orientation term): mismatched finger orientation.
Ablation 7 (i.e., w/o joint position regularization): the hand displays undesired joint configurations.
Real-World Manipulation Tasks
Task 1. Pick up a box, rotate it, and place it down, representing common pick-and-place tasks in dexterous teleoperation.
Full objective
Ablation 1
Ablation 3
All ablations show similar performance in such simple pick-and-place tasks which do not require precise fingertip coordination or accurate contacts.
Task 2. Drag a mug through its handle using the bent thumb, where fingertip orientation plays a decisive role.
Full objective
Ablation 3
Ablation 6
In A3 and A6, no consideration of fingertip orientations results in poor performance as the robot hand fails to replicate the finger bending motion of the human pilots to hook the mug handle.
Task 3. Pick up five different upright-standing screws and place them vertically, where precise pinches are important.
Full objective
Ablation 2
Ablation 3
Using actual pinch distances without rescaling in A2 could lead to low success rate in real-world manipulation tasks, primarily due to that using actual pinch distance is vulnerable to human finger tracking inaccuracy. In A3, the missing of fingertip orientation information also has negative impacts on precise manipulation tasks, as inaccuracy in orientation leads to undesired contact normals with the object.
Results of the real-world manipulation tasks
Task 1 and 2 are assessed by task time, while Task 3 is evaluated by success rate. Each pair of bars shows the error of one ablation implementation and the colors represent the ablation category defined in Table I. The two pilots are distinguished by hatched bars.
Key Takeaways
- Fingertip pinch: removing this term increases positional error and decreases pinch success; crucial for tasks that require precise fingertip coordination.
- Fingertip orientation: without it, the robot hand fails to bend fingers naturally; crucial for tasks sensitive to finger orientation, not just position.
- Global hand pose: using thumb-tip position term (instead of wrist position) allows wrist’s flexible adjustment to improve fingertip accuracy.
- Joint regularization: prevents undesired joint configurations.
- All terms using our proposed formulation demonstrate no conflicts and perform well across all tasks.
BibTeX
@article{xin2026analyzing,
title={Analyzing Key Objectives in Human-to-Robot Retargeting for Dexterous Manipulation},
author={Xin, Chendong and Yu, Mingrui and Jiang, Yongpeng and Zhang, Zhefeng and Li, Xiang},
journal={IEEE Robotics and Automation Practice},
year={2026},
publisher={IEEE}
}