Automatic Physical Examination Segmentation within Objective Structured Clinical Examination Videos

Assessing medical student performance in Objective Structured Clinical Examinations (OSCEs) is labor-intensive, requiring trained evaluators to review 15-minute long videos. The ‘physical examination’ period constitutes only a small portion of these videos. Automated segmentation of OSCE videos could significantly streamline the evaluation process by detecting this physical exam portion for targeted evaluation. Current video analysis approaches struggle with these long recordings due to computational constraints and challenges in maintaining temporal context. This study tests whether multimodal large language models (MM-LLMs) can segment physical examination periods in OSCE videos without prior training, potentially easing the burden on both human graders and automated systems. We analyzed 500 videos from five OSCE stations at UT Southwestern Simulation Center, each 15 minutes long, using hand-labeled physical examination periods as ground truth. MM-LLMs processed video frames at one frame per second, classifying them into discrete activity states. A hidden Markov model with Viterbi decoding ensured temporal consistency across segments, addressing the inherent challenges of frame-by-frame classification. Using Viterbi decoding trained on just 50 hand-labeled videos (10 from each station), zero-shot GPT-4o achieved 99.8% recall and 78.3% intersection over union (IOU), effectively capturing physical examinations with an average duration of 175 seconds from 900-second videos—an 81% reduction in frames requiring review. Integrating multimodal large language models with temporal modeling effectively segments physical examination periods in OSCE videos without requiring extensive training data. This approach significantly reduces review time while maintaining clinical assessment integrity, demonstrating that zero-shot AI methods can be optimized for medical education’s specific requirements. The technique establishes a foundation for more efficient and scalable clinical skills assessment across diverse medical education settings.