YOLOv8 powered deep learning framework for detection and classification of oral potentially malignant disorders and oral cancer using intraoral images.

OBJECTIVE: This study aimed to develop deep learning framework based on the You Only Look Once version 8 (YOLOv8) instance segmentation architecture and to evaluate its performance in detecting and classifying oral potentially malignant disorders (OPMDs) and oral cancer using digital intraoral images. STUDY DESIGN: A dataset comprising 583 clinical intraoral images depicting 227 oral cancer and 356 OPMDs, along with 46 images of clinically healthy oral mucosa as background, was utilized. Following data augmentation techniques, the dataset was expanded to 3,285 images with 1,060 cancer, 2,051 precancer instances and 258 images of healthy oral mucosa. Roboflow was utilized for image classification and annotation, including multi-class labelling and instance segmentation, with models trained by two expert oral pathologists. RESULTS: The YOLOv8 instance segmentation model demonstrated high performance, achieving a mean average precision at a 0.50 mask IoU threshold (mAP@50) of 98.83% and a mAP@50-95 of 78.24%, indicating strong segmentation capabilities. The model also showed excellent classification performance with a Precision of 96.76 and a Recall of 98.24. CONCLUSION: These findings highlight the potential of an AI-driven YOLOv8 segmentation model to support the detection and classification of suspicious oral lesions. The proposed framework may assist general dental practitioners in identifying OPMDs and oral cancer cases that warrant biopsy and facilitate timely referral for appropriate clinical management, particularly in settings with limited access to specialists.