InteractVLM: 3D Interaction Reasoning from 2D Foundational Models
Journal:
arXiv
Published Date:
Apr 7, 2025
Abstract
We introduce InteractVLM, a novel method to estimate 3D contact points on
human bodies and objects from single in-the-wild images, enabling accurate
human-object joint reconstruction in 3D. This is challenging due to occlusions,
depth ambiguities, and widely varying object shapes. Existing methods rely on
3D contact annotations collected via expensive motion-capture systems or
tedious manual labeling, limiting scalability and generalization. To overcome
this, InteractVLM harnesses the broad visual knowledge of large Vision-Language
Models (VLMs), fine-tuned with limited 3D contact data. However, directly
applying these models is non-trivial, as they reason only in 2D, while
human-object contact is inherently 3D. Thus we introduce a novel
Render-Localize-Lift module that: (1) embeds 3D body and object surfaces in 2D
space via multi-view rendering, (2) trains a novel multi-view localization
model (MV-Loc) to infer contacts in 2D, and (3) lifts these to 3D.
Additionally, we propose a new task called Semantic Human Contact estimation,
where human contact predictions are conditioned explicitly on object semantics,
enabling richer interaction modeling. InteractVLM outperforms existing work on
contact estimation and also facilitates 3D reconstruction from an in-the wild
image. Code and models are available at https://interactvlm.is.tue.mpg.de.
