Rethinking Multilingual Vision-Language Translation: Dataset, Evaluation, and Adaptation
Journal:
arXiv
Published Date:
Jun 13, 2025
Abstract
Vision-Language Translation (VLT) is a challenging task that requires
accurately recognizing multilingual text embedded in images and translating it
into the target language with the support of visual context. While recent Large
Vision-Language Models (LVLMs) have demonstrated strong multilingual and visual
understanding capabilities, there is a lack of systematic evaluation and
understanding of their performance on VLT. In this work, we present a
comprehensive study of VLT from three key perspectives: data quality, model
architecture, and evaluation metrics. (1) We identify critical limitations in
existing datasets, particularly in semantic and cultural fidelity, and
introduce AibTrans -- a multilingual, parallel, human-verified dataset with
OCR-corrected annotations. (2) We benchmark 11 commercial LVLMs/LLMs and 6
state-of-the-art open-source models across end-to-end and cascaded
architectures, revealing their OCR dependency and contrasting generation versus
reasoning behaviors. (3) We propose Density-Aware Evaluation to address metric
reliability issues under varying contextual complexity, introducing the DA
Score as a more robust measure of translation quality. Building upon these
findings, we establish a new evaluation benchmark for VLT. Notably, we observe
that fine-tuning LVLMs on high-resource language pairs degrades cross-lingual
performance, and we propose a balanced multilingual fine-tuning strategy that
effectively adapts LVLMs to VLT without sacrificing their generalization
ability.
