Integrating Machine Learning with Musculoskeletal Simulation Improves OpenCap Video-Based Dynamics Estimation

Musculoskeletal dynamics influence the progression and rehabilitation of many movement-related conditions. However, accurately estimating whole-body dynamics using accessible tools, like smartphone video, remains challenging. Physics-based and machine learning (ML)–based dynamic predictions each offer advantages, but both approaches struggle to achieve both high accuracy and physical realism. Here, we created a hybrid ML–simulation framework to improve estimates of ground reaction forces, joint moments, and joint contact forces from smartphone video kinematics. We used machine learning models to predict ground reaction forces and centers of pressure from video-based kinematics. The hybrid framework generates a dynamic simulation that tracks predicted forces and kinematics while enforcing dynamic consistency. We compared the hybrid model’s performance with a simulation-only approach and with ML forces applied through inverse dynamics. We evaluated mean absolute error from lab-based reference data (inverse dynamics from marker and force plate data) from 10 individuals walking. The hybrid model had 29% lower joint moment errors compared to simulations (p<0.001) and 45% lower errors compared to the ML-only approach (p<0.001). It also reduced vertical ground force error by 40% compared to simulations. The hybrid approach improved key metrics of joint loading related to knee osteoarthritis progression by 13–30% compared to simulations. Our hybrid model outperforms purely physics-based and ML approaches for estimating dynamics from smartphone video during walking. These methods move us closer to fast, accurate, and scalable assessments of whole-body musculoskeletal dynamics, which will enable large out-of-lab biomechanics studies and precision treatment of gait-related conditions.