Text-Driven Diffusion Model for Sign Language Production
Journal:
arXiv
Published Date:
Mar 20, 2025
Abstract
We introduce the hfut-lmc team's solution to the SLRTP Sign Production
Challenge. The challenge aims to generate semantically aligned sign language
pose sequences from text inputs. To this end, we propose a Text-driven
Diffusion Model (TDM) framework. During the training phase, TDM utilizes an
encoder to encode text sequences and incorporates them into the diffusion model
as conditional input to generate sign pose sequences. To guarantee the high
quality and accuracy of the generated pose sequences, we utilize two key loss
functions. The joint loss function L_{joint} is used to precisely measure and
minimize the differences between the joint positions of the generated pose
sequences and those of the ground truth. Similarly, the bone orientation loss
function L_{bone} is instrumental in ensuring that the orientation of the bones
in the generated poses aligns with the actual, correct orientations. In the
inference stage, the TDM framework takes on a different yet equally important
task. It starts with noisy sequences and, under the strict constraints of the
text conditions, gradually refines and generates semantically consistent sign
language pose sequences. Our carefully designed framework performs well on the
sign language production task, and our solution achieves a BLEU-1 score of
20.17, placing second in the challenge.
