Deep Learning-Based Semantic Segmentation for Real-Time Kidney Imaging and Measurements with Augmented Reality-Assisted Ultrasound
Journal:
arXiv
Published Date:
Jun 30, 2025
Abstract
Ultrasound (US) is widely accessible and radiation-free but has a steep
learning curve due to its dynamic nature and non-standard imaging planes.
Additionally, the constant need to shift focus between the US screen and the
patient poses a challenge. To address these issues, we integrate deep learning
(DL)-based semantic segmentation for real-time (RT) automated kidney volumetric
measurements, which are essential for clinical assessment but are traditionally
time-consuming and prone to fatigue. This automation allows clinicians to
concentrate on image interpretation rather than manual measurements.
Complementing DL, augmented reality (AR) enhances the usability of US by
projecting the display directly into the clinician's field of view, improving
ergonomics and reducing the cognitive load associated with screen-to-patient
transitions. Two AR-DL-assisted US pipelines on HoloLens-2 are proposed: one
streams directly via the application programming interface for a wireless
setup, while the other supports any US device with video output for broader
accessibility. We evaluate RT feasibility and accuracy using the Open Kidney
Dataset and open-source segmentation models (nnU-Net, Segmenter, YOLO with
MedSAM and LiteMedSAM). Our open-source GitHub pipeline includes model
implementations, measurement algorithms, and a Wi-Fi-based streaming solution,
enhancing US training and diagnostics, especially in point-of-care settings.
