Artificial intelligence-powered 3D analysis of video-based caregiver-child interactions.

Journal: Science advances
PMID: 39951536

Abstract

We introduce HARMONI, a three-dimensional (3D) computer vision and audio processing method for analyzing caregiver-child behavior and interaction from observational videos. HARMONI operates at subsecond resolution, estimating 3D mesh representations and spatial interactions of humans, and adapts to challenging natural environments using an environment-targeted synthetic data generation module. Deployed on 500 hours from the SEEDLingS dataset, HARMONI generates detailed quantitative measurements of 3D human behavior previously unattainable through manual efforts or 2D methods. HARMONI identifies longitudinal trends in child-caregiver interaction, including child movement, body pose, dyadic touch, visibility, and conversational turns. The integrated visual and audio analysis further reveals multimodal trends, including associations between child conversational turns and movement. Open-sourced for large-scale analysis, HARMONI facilitates advancements in human development research. HARMONI achieves 63 to 80% consistency on key attributes with human annotators on SEEDLingS and 84 to 93% consistency on videos taken from a laboratory setting while achieving >100 times savings in time.

Authors

  • Zhenzhen Weng
    Institute for Computational & Mathematical Engineering, Stanford University, Stanford, CA, USA.
  • Laura Bravo-Sánchez
    Department of Biomedical Data Science, Stanford University, Stanford, CA, USA.
  • Zeyu Wang
    Department of Neurosurgery, Xiangya Hospital, Central South University, Changsha, China.
  • Christopher Howard
    Department of Computer Science, Stanford University, Stanford, CA, USA.
  • Maria Xenochristou
    Stanford University, Stanford, CA, USA.
  • Nicole Meister
    Department of Electrical Engineering, Stanford University, Stanford, CA, USA.
  • Angjoo Kanazawa
    Department of Electrical Engineering and Computer Sciences, UC Berkeley, Berkeley, CA, USA.
  • Arnold Milstein
    Stanford Clinical Excellence Research Center, Stanford University, Stanford, CA, USA.
  • Elika Bergelson
    Department of Psychology, Harvard University, Cambridge, MA, USA.
  • Kathryn L Humphreys
    Department of Psychology and Human Development, Vanderbilt University, Nashville, TN, USA.
  • Lee M Sanders
    Pediatrics and Health Policy, Stanford University, Stanford, CA, USA.
  • Serena Yeung-Levy
    Stanford University, Stanford, CA, USA.

Keywords

External Resources

Popular Topics
Recent Journals