Contrastive learning-based fusion of NIR spectroscopy and visual cues for molecular discrimination of Chinese herbal species.

Accurate identification of Chinese medicinal herbs is essential for ensuring quality control and clinical safety. However, conventional visual inspection and machine vision techniques often struggle to distinguish morphologically similar herbs. This study proposes a cross-modal neural framework, named as CNRP, which integrates RGB imaging and near-infrared (NIR) spectroscopy through contrastive learning to enhance fine-grained herbal classification. CNRP is designed as a dual-stream architecture, which extracts key features from RGB images and NIR spectra through image encoder and spectral encoder respectively, then enhances the global information in the features through the multi-head attention modules, and finally performs semantic alignment of the dual stream data features. RGB and NIR are used simultaneously to train the CNRP network and construct the NIR feature set; in the inference stage, it is only necessary to extract features from the input RGB image and match the features with similar features in the NIR feature set to complete the classification of the input RGB image. Experimental results on 8 categories of visually confusing herb demonstrate that the proposed method achieves overall accuracy of 97.32 %, which is at least 16.11 % higher than the conventional vision baselines. This work provides a cost-effective and scalable solution for intelligent herbal authentication.