Efficient Medical VIE via Reinforcement Learning
Journal:
arXiv
Published Date:
Jun 16, 2025
Abstract
Visual Information Extraction (VIE) converts unstructured document images
into structured formats like JSON, critical for medical applications such as
report analysis and online consultations. Traditional methods rely on OCR and
language models, while end-to-end multimodal models offer direct JSON
generation. However, domain-specific schemas and high annotation costs limit
their effectiveness in medical VIE. We base our approach on the Reinforcement
Learning with Verifiable Rewards (RLVR) framework to address these challenges
using only 100 annotated samples. Our approach ensures dataset diversity, a
balanced precision-recall reward mechanism to reduce hallucinations and improve
field coverage, and innovative sampling strategies to enhance reasoning
capabilities. Fine-tuning Qwen2.5-VL-7B with our RLVR method, we achieve
state-of-the-art performance on medical VIE tasks, significantly improving F1,
precision, and recall. While our models excel on tasks similar to medical
datasets, performance drops on dissimilar tasks, highlighting the need for
domain-specific optimization. Case studies further demonstrate the value of
reasoning during training and inference for VIE.
