PromptHMR: Promptable Human Mesh Recovery
Journal:
arXiv
Published Date:
Apr 8, 2025
Abstract
Human pose and shape (HPS) estimation presents challenges in diverse
scenarios such as crowded scenes, person-person interactions, and single-view
reconstruction. Existing approaches lack mechanisms to incorporate auxiliary
"side information" that could enhance reconstruction accuracy in such
challenging scenarios. Furthermore, the most accurate methods rely on cropped
person detections and cannot exploit scene context while methods that process
the whole image often fail to detect people and are less accurate than methods
that use crops. While recent language-based methods explore HPS reasoning
through large language or vision-language models, their metric accuracy is well
below the state of the art. In contrast, we present PromptHMR, a
transformer-based promptable method that reformulates HPS estimation through
spatial and semantic prompts. Our method processes full images to maintain
scene context and accepts multiple input modalities: spatial prompts like
bounding boxes and masks, and semantic prompts like language descriptions or
interaction labels. PromptHMR demonstrates robust performance across
challenging scenarios: estimating people from bounding boxes as small as faces
in crowded scenes, improving body shape estimation through language
descriptions, modeling person-person interactions, and producing temporally
coherent motions in videos. Experiments on benchmarks show that PromptHMR
achieves state-of-the-art performance while offering flexible prompt-based
control over the HPS estimation process.
