H2R: A Human-to-Robot Data Augmentation for Robot Pre-training from Videos
Journal:
arXiv
Published Date:
May 17, 2025
Abstract
Large-scale pre-training using videos has proven effective for robot
learning. However, the models pre-trained on such data can be suboptimal for
robot learning due to the significant visual gap between human hands and those
of different robots. To remedy this, we propose H2R, a simple data augmentation
technique that detects human hand keypoints, synthesizes robot motions in
simulation, and composites rendered robots into egocentric videos. This process
explicitly bridges the visual gap between human and robot embodiments during
pre-training. We apply H2R to augment large-scale egocentric human video
datasets such as Ego4D and SSv2, replacing human hands with simulated robotic
arms to generate robot-centric training data. Based on this, we construct and
release a family of 1M-scale datasets covering multiple robot embodiments (UR5
with gripper/Leaphand, Franka) and data sources (SSv2, Ego4D). To verify the
effectiveness of the augmentation pipeline, we introduce a CLIP-based
image-text similarity metric that quantitatively evaluates the semantic
fidelity of robot-rendered frames to the original human actions. We validate
H2R across three simulation benchmarks: Robomimic, RLBench and PushT and
real-world manipulation tasks with a UR5 robot equipped with Gripper and
Leaphand end-effectors. H2R consistently improves downstream success rates,
yielding gains of 5.0%-10.2% in simulation and 6.7%-23.3% in real-world tasks
across various visual encoders and policy learning methods. These results
indicate that H2R improves the generalization ability of robotic policies by
mitigating the visual discrepancies between human and robot domains.
