Towards Dynamic Neural Communication and Speech Neuroprosthesis Based on Viseme Decoding
Journal:
arXiv
Published Date:
Jan 9, 2025
Abstract
Decoding text, speech, or images from human neural signals holds promising
potential both as neuroprosthesis for patients and as innovative communication
tools for general users. Although neural signals contain various information on
speech intentions, movements, and phonetic details, generating informative
outputs from them remains challenging, with mostly focusing on decoding short
intentions or producing fragmented outputs. In this study, we developed a
diffusion model-based framework to decode visual speech intentions from
speech-related non-invasive brain signals, to facilitate face-to-face neural
communication. We designed an experiment to consolidate various phonemes to
train visemes of each phoneme, aiming to learn the representation of
corresponding lip formations from neural signals. By decoding visemes from both
isolated trials and continuous sentences, we successfully reconstructed
coherent lip movements, effectively bridging the gap between brain signals and
dynamic visual interfaces. The results highlight the potential of viseme
decoding and talking face reconstruction from human neural signals, marking a
significant step toward dynamic neural communication systems and speech
neuroprosthesis for patients.
