How do Foundation Models Compare to Skeleton-Based Approaches for Gesture Recognition in Human-Robot Interaction?
Journal:
arXiv
Published Date:
Jun 25, 2025
Abstract
Gestures enable non-verbal human-robot communication, especially in noisy
environments like agile production. Traditional deep learning-based gesture
recognition relies on task-specific architectures using images, videos, or
skeletal pose estimates as input. Meanwhile, Vision Foundation Models (VFMs)
and Vision Language Models (VLMs) with their strong generalization abilities
offer potential to reduce system complexity by replacing dedicated
task-specific modules. This study investigates adapting such models for
dynamic, full-body gesture recognition, comparing V-JEPA (a state-of-the-art
VFM), Gemini Flash 2.0 (a multimodal VLM), and HD-GCN (a top-performing
skeleton-based approach). We introduce NUGGET, a dataset tailored for
human-robot communication in intralogistics environments, to evaluate the
different gesture recognition approaches. In our experiments, HD-GCN achieves
best performance, but V-JEPA comes close with a simple, task-specific
classification head - thus paving a possible way towards reducing system
complexity, by using it as a shared multi-task model. In contrast, Gemini
struggles to differentiate gestures based solely on textual descriptions in the
zero-shot setting, highlighting the need of further research on suitable input
representations for gestures.
