Pose as a Modality: A Psychology-Inspired Network for Personality Recognition with a New Multimodal Dataset
Journal:
arXiv
Published Date:
Mar 17, 2025
Abstract
In recent years, predicting Big Five personality traits from multimodal data
has received significant attention in artificial intelligence (AI). However,
existing computational models often fail to achieve satisfactory performance.
Psychological research has shown a strong correlation between pose and
personality traits, yet previous research has largely ignored pose data in
computational models. To address this gap, we develop a novel multimodal
dataset that incorporates full-body pose data. The dataset includes video
recordings of 287 participants completing a virtual interview with 36
questions, along with self-reported Big Five personality scores as labels. To
effectively utilize this multimodal data, we introduce the Psychology-Inspired
Network (PINet), which consists of three key modules: Multimodal Feature
Awareness (MFA), Multimodal Feature Interaction (MFI), and Psychology-Informed
Modality Correlation Loss (PIMC Loss). The MFA module leverages the Vision
Mamba Block to capture comprehensive visual features related to personality,
while the MFI module efficiently fuses the multimodal features. The PIMC Loss,
grounded in psychological theory, guides the model to emphasize different
modalities for different personality dimensions. Experimental results show that
the PINet outperforms several state-of-the-art baseline models. Furthermore,
the three modules of PINet contribute almost equally to the model's overall
performance. Incorporating pose data significantly enhances the model's
performance, with the pose modality ranking mid-level in importance among the
five modalities. These findings address the existing gap in personality-related
datasets that lack full-body pose data and provide a new approach for improving
the accuracy of personality prediction models, highlighting the importance of
integrating psychological insights into AI frameworks.
