Evaluating human-AI interaction in optical diagnosis of early colorectal carcinoma through endoscopist responses to simulated artificial intelligence: a multinational online study.

Background Suboptimal human - artificial intelligence (AI) interaction is a potential roadblock for implementation of AI in clinical practice. We aimed to evaluate interaction between AI and endoscopists in optical diagnosis of colorectal carcinoma (CRC). Methods International endoscopists were invited to diagnose colorectal lesions online. After a pretest of 15 videos, the diagnosis could be adjusted in the next 35 videos after receiving a simulated AI diagnosis. Histology was the gold standard. Endoscopist's diagnoses before and after AI were compared using the signal detection theory. Inappropriate AI response meant decreased diagnostic performance by the endoscopist after AI. Results In 2730 diagnoses, the 78 participants reached diagnostic accuracies of 71.4% before and 73.7% after AI (p<0.001). Before AI, endoscopists missed only few true CRCs (mean hit rate 0.86), at the cost of a mean false alarm rate of 0.35. Inappropriate AI response was observed in 28.2% of the participants. After AI, most endoscopists (41.0%) shifted towards a more conservative attitude, tending to diagnose a lesion as non-CRC. Conclusions Inappropriate AI response is an issue in human-AI interaction for the optical diagnosis of CRCs. Before using AI, endoscopists could inform themselves on their own and AI's optical diagnostic performance to possibly improve interaction.