MMFVS-Net: A triple-symmetric cross-attention network for multimodal optical image fusion and high-accuracy virtual staining of breast cancer tissues.

BACKGROUND AND OBJECTIVE: Virtual staining is based on label-free imaging technology and deep learning, which enables the efficient and non-destructive generation of virtual stained images with effects similar to those of traditional pathological staining, thus holding great application potential in future clinical medical practice. However, the information provided by a single imaging technology is limited, which is insufficient to support high-accuracy virtual staining. METHODS: This study took breast cancer tissue as the research object. Bright-field, auto-fluorescence and phase contrast imaging technology were introduced to simultaneously extract the multimodal complementary information. A multi-branch input network was established to fuse the three types of optical images and generate high-accuracy virtual stained images. To enhance the model's performance, three strategies were employed, including designing a Triple-symmetric cross-attention module to handle the interactive information of multimodal image features, introducing multi-scale discriminator to ensure the model's ability to capture both global features and local details, and applying transfer-learning to address the gradient dilution issue of multi-branches. RESULTS: Through the evaluation of various image similarity metrics and nuclear statistical metrics, the virtual stained images based on multimodal fusion achieved high accuracy and were significantly superior to any single-modal virtual staining approach. Further ablation experiments explained the reason of the aforementioned results and quantitatively demonstrated the improvements brought by the three optimization strategies. CONCLUSIONS: This work advances the application of multimodal fusion in virtual staining, address the key challenges of cross-modal interactive information mining and multi-branch model training, and further enhance the accuracy and reliability of virtual staining.