A Speech-to-Video Synthesis Approach Using Spatio-Temporal Diffusion for Vocal Tract MRI
Journal:
arXiv
Published Date:
Mar 15, 2025
Abstract
Understanding the relationship between vocal tract motion during speech and
the resulting acoustic signal is crucial for aided clinical assessment and
developing personalized treatment and rehabilitation strategies. Toward this
goal, we introduce an audio-to-video generation framework for creating Real
Time/cine-Magnetic Resonance Imaging (RT-/cine-MRI) visuals of the vocal tract
from speech signals. Our framework first preprocesses RT-/cine-MRI sequences
and speech samples to achieve temporal alignment, ensuring synchronization
between visual and audio data. We then employ a modified stable diffusion
model, integrating structural and temporal blocks, to effectively capture
movement characteristics and temporal dynamics in the synchronized data. This
process enables the generation of MRI sequences from new speech inputs,
improving the conversion of audio into visual data. We evaluated our framework
on healthy controls and tongue cancer patients by analyzing and comparing the
vocal tract movements in synthesized videos. Our framework demonstrated
adaptability to new speech inputs and effective generalization. In addition,
positive human evaluations confirmed its effectiveness, with realistic and
accurate visualizations, suggesting its potential for outpatient therapy and
personalized simulation of vocal tract visualizations.
