Towards Scalable and Cross-Lingual Specialist Language Models for Oncology
Journal:
arXiv
Published Date:
Mar 11, 2025
Abstract
Clinical oncology generates vast, unstructured data that often contain
inconsistencies, missing information, and ambiguities, making it difficult to
extract reliable insights for data-driven decision-making. General-purpose
large language models (LLMs) struggle with these challenges due to their lack
of domain-specific reasoning, including specialized clinical terminology,
context-dependent interpretations, and multi-modal data integration. We address
these issues with an oncology-specialized, efficient, and adaptable NLP
framework that combines instruction tuning, retrieval-augmented generation
(RAG), and graph-based knowledge integration. Our lightweight models prove
effective at oncology-specific tasks, such as named entity recognition (e.g.,
identifying cancer diagnoses), entity linking (e.g., linking entities to
standardized ontologies), TNM staging, document classification (e.g., cancer
subtype classification from pathology reports), and treatment response
prediction. Our framework emphasizes adaptability and resource efficiency. We
include minimal German instructions, collected at the University Hospital
Zurich (USZ), to test whether small amounts of non-English language data can
effectively transfer knowledge across languages. This approach mirrors our
motivation for lightweight models, which balance strong performance with
reduced computational costs, making them suitable for resource-limited
healthcare settings. We validated our models on oncology datasets,
demonstrating strong results in named entity recognition, relation extraction,
and document classification.
