Streamlining the Abstract Selection Process for the American Society of Breast Surgeons' Annual Meeting by Utilizing Large Language Models.

PURPOSE: The American Society of Breast Surgeons (ASBrS) receives an increasing number of abstracts for its annual meeting, resulting in increasing time and effort for peer-review. We aimed to compare artificial intelligence (AI) large language models (LLMs) to human-generated scores to explore if AI-review is a plausible abstract screening process. METHODS: Abstracts published from the 2025 ASBrS meeting were assessed with three LLMs: OpenAI's GPT4-o (GPT), Meta's Llama-3.1-405b (Llama) and DeepSeek's DeepSeek-V3 (DeepSeek). Abstracts were analyzed using zero shot (ZS), which included the 2025 ASBrS scoring rubric and few shot (FS) with examples of high- and low-scoring abstracts. Large language model scores were compared with the committee's scores. Model inference was evaluated by using Spearman rank correlation, quartile accuracy, and mean absolute error; 95% CIs were calculated by using bootstrap resampling. RESULTS: A total of 378 published accepted abstracts were included. Human-generated median abstract score was 21.2 (rubric ranged 0-35). Median scores were higher amongst all LLMs for both the ZS and FS approaches (27.0-29.7, p < 0.001). Spearman values with the ZS approach for GPT, DeepSeek, and Llama were 0.24, 0.30, and 0.31, respectively, and for the FS approach were 0.37, 0.35, and 0.27, respectively (p < 0.001 for all). DeepSeek ranked the first quartile correctly with 53.5% accuracy while Llama ranked the fourth quartile correctly with 42.9% accuracy. The FS approach modestly improved correlation and first quartile accuracy, with greatest improvement seen in GPT. CONCLUSIONS: Large language models showed statistically significant rank correlation with human ASBrS abstract ranking. Large language models may be useful as a preliminary screening tool to prioritize reviewer effort toward higher-quality abstracts.