MT$^{3}$: Scaling MLLM-based Text Image Machine Translation via Multi-Task Reinforcement Learning
Journal:
arXiv
Published Date:
May 26, 2025
Abstract
Text Image Machine Translation (TIMT)-the task of translating textual content
embedded in images-is critical for applications in accessibility, cross-lingual
information access, and real-world document understanding. However, TIMT
remains a complex challenge due to the need for accurate optical character
recognition (OCR), robust visual-text reasoning, and high-quality translation,
often requiring cascading multi-stage pipelines. Recent advances in large-scale
Reinforcement Learning (RL) have improved reasoning in Large Language Models
(LLMs) and Multimodal LLMs (MLLMs), but their application to end-to-end TIMT is
still underexplored. To bridge this gap, we introduce MT$^{3}$, the first
framework to apply Multi-Task RL to MLLMs for end-to-end TIMT. MT$^{3}$ adopts
a multi-task optimization paradigm targeting three key sub-skills: text
recognition, context-aware reasoning, and translation. It is trained using a
novel multi-mixed reward mechanism that adapts rule-based RL strategies to
TIMT's intricacies, offering fine-grained, non-binary feedback across tasks.
Furthermore, to facilitate the evaluation of TIMT in authentic cross-cultural
and real-world social media contexts, we introduced XHSPost, the first social
media TIMT benchmark. Our MT$^{3}$-7B-Zero achieves state-of-the-art results on
the latest in-domain MIT-10M benchmark, outperforming strong baselines such as
Qwen2.5-VL-72B and InternVL2.5-78B by notable margins across multiple metrics.
Additionally, the model shows strong generalization to out-of-distribution
language pairs and datasets. In-depth analyses reveal how multi-task synergy,
reinforcement learning initialization, curriculum design, and reward
formulation contribute to advancing MLLM-driven TIMT.
