HSENet: Hybrid Spatial Encoding Network for 3D Medical Vision-Language Understanding
Journal:
arXiv
Published Date:
Jun 11, 2025
Abstract
Automated 3D CT diagnosis empowers clinicians to make timely, evidence-based
decisions by enhancing diagnostic accuracy and workflow efficiency. While
multimodal large language models (MLLMs) exhibit promising performance in
visual-language understanding, existing methods mainly focus on 2D medical
images, which fundamentally limits their ability to capture complex 3D
anatomical structures. This limitation often leads to misinterpretation of
subtle pathologies and causes diagnostic hallucinations. In this paper, we
present Hybrid Spatial Encoding Network (HSENet), a framework that exploits
enriched 3D medical visual cues by effective visual perception and projection
for accurate and robust vision-language understanding. Specifically, HSENet
employs dual-3D vision encoders to perceive both global volumetric contexts and
fine-grained anatomical details, which are pre-trained by dual-stage alignment
with diagnostic reports. Furthermore, we propose Spatial Packer, an efficient
multimodal projector that condenses high-resolution 3D spatial regions into a
compact set of informative visual tokens via centroid-based compression. By
assigning spatial packers with dual-3D vision encoders, HSENet can seamlessly
perceive and transfer hybrid visual representations to LLM's semantic space,
facilitating accurate diagnostic text generation. Experimental results
demonstrate that our method achieves state-of-the-art performance in 3D
language-visual retrieval (39.85% of R@100, +5.96% gain), 3D medical report
generation (24.01% of BLEU-4, +8.01% gain), and 3D visual question answering
(73.60% of Major Class Accuracy, +1.99% gain), confirming its effectiveness.
Our code is available at https://github.com/YanzhaoShi/HSENet.
