Bootstrapping Grounded Chain-of-Thought in Multimodal LLMs for Data-Efficient Model Adaptation
Journal:
arXiv
Published Date:
Jul 3, 2025
Abstract
Multimodal Large Language Models (MLLMs) have demonstrated remarkable
capabilities in interpreting images using natural language. However, without
using large-scale datasets for retraining, these models are difficult to adapt
to specialized vision tasks, e.g., chart understanding. This problem is caused
by a mismatch between pre-training and downstream datasets: pre-training
datasets primarily concentrate on scenes and objects but contain limited
information about specialized, non-object images, such as charts and tables. In
this paper, we share an interesting finding that training an MLLM with
Chain-of-Thought (CoT) reasoning data can facilitate model adaptation in
specialized vision tasks, especially under data-limited regimes. However, we
identify a critical issue within CoT data distilled from pre-trained MLLMs,
i.e., the data often contains multiple factual errors in the reasoning steps.
To address the problem, we propose Grounded Chain-of-Thought (GCoT), a simple
bootstrapping-based approach that aims to inject grounding information (i.e.,
bounding boxes) into CoT data, essentially making the reasoning steps more
faithful to input images. We evaluate our approach on five specialized vision
tasks, which cover a variety of visual formats including charts, tables,
receipts, and reports. The results demonstrate that under data-limited regimes
our approach significantly improves upon fine-tuning and distillation.
