Improving MLLM's Document Image Machine Translation via Synchronously Self-reviewing Its OCR Proficiency
Journal:
arXiv
Published Date:
Jul 11, 2025
Abstract
Multimodal Large Language Models (MLLMs) have shown strong performance in
document image tasks, especially Optical Character Recognition (OCR). However,
they struggle with Document Image Machine Translation (DIMT), which requires
handling both cross-modal and cross-lingual challenges. Previous efforts to
enhance DIMT capability through Supervised Fine-Tuning (SFT) on the DIMT
dataset often result in the forgetting of the model's existing monolingual
abilities, such as OCR. To address these challenges, we introduce a novel
fine-tuning paradigm, named Synchronously Self-Reviewing (SSR) its OCR
proficiency, inspired by the concept "Bilingual Cognitive Advantage".
Specifically, SSR prompts the model to generate OCR text before producing
translation text, which allows the model to leverage its strong monolingual OCR
ability while learning to translate text across languages. Comprehensive
experiments demonstrate the proposed SSR learning helps mitigate catastrophic
forgetting, improving the generalization ability of MLLMs on both OCR and DIMT
tasks.
