X-RAFT: Cross-Modal Non-Rigid Registration of Blue and White Light Neurosurgical Hyperspectral Images
Journal:
arXiv
Published Date:
Jul 10, 2025
Abstract
Integration of hyperspectral imaging into fluorescence-guided neurosurgery
has the potential to improve surgical decision making by providing quantitative
fluorescence measurements in real-time. Quantitative fluorescence requires
paired spectral data in fluorescence (blue light) and reflectance (white light)
mode. Blue and white image acquisition needs to be performed sequentially in a
potentially dynamic surgical environment. A key component to the fluorescence
quantification process is therefore the ability to find dense cross-modal image
correspondences between two hyperspectral images taken under these drastically
different lighting conditions. We address this challenge with the introduction
of X-RAFT, a Recurrent All-Pairs Field Transforms (RAFT) optical flow model
modified for cross-modal inputs. We propose using distinct image encoders for
each modality pair, and fine-tune these in a self-supervised manner using
flow-cycle-consistency on our neurosurgical hyperspectral data. We show an
error reduction of 36.6% across our evaluation metrics when comparing to a
naive baseline and 27.83% reduction compared to an existing cross-modal optical
flow method (CrossRAFT). Our code and models will be made publicly available
after the review process.
