$μ^2$Tokenizer: Differentiable Multi-Scale Multi-Modal Tokenizer for Radiology Report Generation
Journal:
arXiv
Published Date:
Jun 30, 2025
Abstract
Automated radiology report generation (RRG) aims to produce detailed textual
reports from clinical imaging, such as computed tomography (CT) scans, to
improve the accuracy and efficiency of diagnosis and provision of management
advice. RRG is complicated by two key challenges: (1) inherent complexity in
extracting relevant information from imaging data under resource constraints,
and (2) difficulty in objectively evaluating discrepancies between
model-generated and expert-written reports. To address these challenges, we
propose $\mu^2$LLM, a $\underline{\textbf{mu}}$ltiscale
$\underline{\textbf{mu}}$ltimodal large language models for RRG tasks. The
novel ${\mu}^2$Tokenizer, as an intermediate layer, integrates multi-modal
features from the multiscale visual tokenizer and the text tokenizer, then
enhances report generation quality through direct preference optimization
(DPO), guided by GREEN-RedLlama. Experimental results on four large CT
image-report medical datasets demonstrate that our method outperforms existing
approaches, highlighting the potential of our fine-tuned $\mu^2$LLMs on limited
data for RRG tasks. At the same time, for prompt engineering, we introduce a
five-stage, LLM-driven pipeline that converts routine CT reports into paired
visual-question-answer triples and citation-linked reasoning narratives,
creating a scalable, high-quality supervisory corpus for explainable multimodal
radiology LLM. All code, datasets, and models will be publicly available in our
official repository. https://github.com/Siyou-Li/u2Tokenizer
