Text-Driven Tumor Synthesis
Journal:
arXiv
Published Date:
Dec 24, 2024
Abstract
Tumor synthesis can generate examples that AI often misses or over-detects,
improving AI performance by training on these challenging cases. However,
existing synthesis methods, which are typically unconditional -- generating
images from random variables -- or conditioned only by tumor shapes, lack
controllability over specific tumor characteristics such as texture,
heterogeneity, boundaries, and pathology type. As a result, the generated
tumors may be overly similar or duplicates of existing training data, failing
to effectively address AI's weaknesses. We propose a new text-driven tumor
synthesis approach, termed TextoMorph, that provides textual control over tumor
characteristics. This is particularly beneficial for examples that confuse the
AI the most, such as early tumor detection (increasing Sensitivity by +8.5%),
tumor segmentation for precise radiotherapy (increasing DSC by +6.3%), and
classification between benign and malignant tumors (improving Sensitivity by
+8.2%). By incorporating text mined from radiology reports into the synthesis
process, we increase the variability and controllability of the synthetic
tumors to target AI's failure cases more precisely. Moreover, TextoMorph uses
contrastive learning across different texts and CT scans, significantly
reducing dependence on scarce image-report pairs (only 141 pairs used in this
study) by leveraging a large corpus of 34,035 radiology reports. Finally, we
have developed rigorous tests to evaluate synthetic tumors, including
Text-Driven Visual Turing Test and Radiomics Pattern Analysis, showing that our
synthetic tumors is realistic and diverse in texture, heterogeneity,
boundaries, and pathology.
