Development and initial validation of a video-mediated listening test based on multimodal generative AI.

Video-mediated listening has been recognized for its potential to facilitate learners' comprehension, engagement, and motivation by providing richer contextual cues and audiovisual input. However, the development of authentic listening assessments typically requires substantial time and financial resources, which hinders their practical implementation. This study aims to address these challenges by applying generative AI to develop a video-mediated listening assessment. Multimodal generative AI (InVideo) was employed to create four videos, while GPT-4 was used to automatically generate 20 test items targeting five listening subskills. A total of 542 university students completed the AI-based video-mediated listening assessment within 30 min, followed by a questionnaire. Additionally, five students and five teachers were selected for semi-structured interviews. Results from psychometric models showed that 19 out of 20 items effectively measured five targeted listening subskills, with appropriate item difficulty and discrimination, confirming construct validity. The questionnaire suggested that the AI-generated videos were adequate in terms of audio, visual, and audio-visual consistency, providing support for the relevance and utility of the test. Questionnaires and semi-structured interviews indicated that video-mediated listening seemed to improve students' listening ability and enhance their interest, motivation, and engagement compared to audio-based methods, while teachers reported that multimodal AI reduced the emotional labor of preparing audiovisual materials in language teaching, supporting the assessment's positive consequences. The fundamental validity considerations of AI-based video-mediated listening assessment and implications for language learning and teaching were discussed.