Beyond Speaker Identity: Text Guided Target Speech Extraction
Journal:
arXiv
Published Date:
Jan 15, 2025
Abstract
Target Speech Extraction (TSE) traditionally relies on explicit clues about
the speaker's identity like enrollment audio, face images, or videos, which may
not always be available. In this paper, we propose a text-guided TSE model
StyleTSE that uses natural language descriptions of speaking style in addition
to the audio clue to extract the desired speech from a given mixture. Our model
integrates a speech separation network adapted from SepFormer with a
bi-modality clue network that flexibly processes both audio and text clues. To
train and evaluate our model, we introduce a new dataset TextrolMix with speech
mixtures and natural language descriptions. Experimental results demonstrate
that our method effectively separates speech based not only on who is speaking,
but also on how they are speaking, enhancing TSE in scenarios where traditional
audio clues are absent. Demos are at:
https://mingyue66.github.io/TextrolMix/demo/
