YOLO-spectra: A generalized framework for rapid simultaneous detection and classification of Raman spectra in images with mobile devices for enhancing on-site applications.

BACKGROUND: Implementing deep learning (DL)-based computer vision models for spectroscopy enables real-time, direct spectral classification during measurement by capturing images through smartphones, bypassing data collection and pre-processing associated with traditional machine learning (ML) methods. The main difficulty lies in acquiring balanced training spectral data and creation of diverse annotated image dataset for robust DL model training. Therefore, efficient methods for spectral data collection and creation of annotated image datasets are necessary for accurately identifying analytes and mixtures, facilitating intelligent spectroscopic solutions. RESULTS: This study introduces the YOLO (You Only Look Once) model for analysis of diverse spectra of pharmaceutical mixtures, trace pesticides and dyes, prepared with a cost-effective, simple-to-use surface-enhanced Raman spectroscopy technique combined with a portable spectrometer towards affordable customized applications. A novel scalable vector graphics (SVG) approach is presented for automated production of large-scale annotated spectral image datasets, accommodating various spectrometer display settings. The YOLOv8m model showed exceptional mean average precision (mAP50) of 0.992 and mAP50-95 over 0.991 for classification of 90 different spectra classes comprising of open-source solvents dataset and experimentally generated Raman/SERS dataset including different composition varied mixtures. These metrics were robust against diverse variations of spectral shifts, intensities, noise, fluorescence background, and spectral range, surpassing 1D CNN analysis. YOLOv8m performance is benchmarked against a smaller YOLOv8n model and its wide applicability is shown with precise identification of FTIR spectra with eight different representations. With the model training complemented by albumentations such as skew and rotation, we demonstrated precise spectra detection and classification via smartphone by capturing the images from computer screen or printed paper. SIGNIFICANCE: For spectroscopy, the YOLO method provides an alternative approach for spectra classification with several aspects: 1) data length or spectral range independent quick analysis with spectra visuals, 2) on-spot identification of multiple spectra in a single image, and 3) simplified analysis procedure from a trained model on analytes of interest. Moreover, YOLO has demonstrated reliable identification of compound mixtures of fixed compositions from spectra of closely varied features that fulfill the main aim of spectroscopy for wide range of applications.