Assisting Screening of Adolescent Depression With Large Language Models as Symptom Detectors: Algorithm Development and Validation.

BACKGROUND: Depression prevalence is rising in adolescents and young adults (ages 10-24 years). Traditional screening methods for depression, such as the Patient Health Questionnaire (PHQ-9), are particularly challenging for adolescents. AI has potential to help with this, but the scarcity of annotated datasets highlights the need for off-the-shelf approaches. In this study, we investigate the feasibility of using state-of-the-art large language models (LLMs) to detect depressive symptoms in adolescent settings (ages 15-17 years). OBJECTIVE: This study aimed to evaluate the feasibility of using in-context learning with LLMs to identify depressive symptoms in free-text clinical notes for adolescent populations, to evaluate the feasibility of off-the-shelf LLM configurations in detecting PHQ-9 symptom-related information under zero- and few-shot settings, and to demonstrate the potential clinical utility of LLM-driven evidence extraction through an interpretable AI-based depression screening tool. METHODS: We examined free text of the electronic health records (EHRs) of adolescent patients with the diagnosis of depression or related mood disorders (age groups 15-17 years, 924 patients) from the Cincinnati Children's Hospital Medical Center. We leveraged the combination of the Beck Depression Inventory (BDI) and the PHQ-9 to develop tailored categories suited for adolescent depression symptoms. We applied 3 state-of-the-art LLMs (Llama, smaller FLAN-T5, and Phi) to identify these symptom categories in zero- and few-shot configurations, allowing us to assess their performance without additional task-specific training. A word-search model was used as the baseline comparator. RESULTS: We manually annotated clinical notes for 22 patients across 16 depression-related symptom categories. Our analysis shows that all LLMs substantially outperformed the word-matching baseline. Within the evaluated prompting configurations, FLAN-T5 achieved the highest precision (average F1-score=0.65, precision=0.78) and strong performance in detecting rare symptoms such as sleep problems (F1-score=0.92) and self-loathing (F1-score=0.80). Phi offered a balanced performance (precision=0.44, recall=0.60), while Llama 3 achieved the highest recall (0.90) but tended to overgeneralize, making it less suitable for precise screening. Within this pilot study, the best-performing LLM exceeded the baseline F1-score by 0.43. Finally, we demonstrate the clinical utility of FLAN-T5 annotations as features in a machine learning model, which distinguished depression cases from controls with high precision (0.78). CONCLUSIONS: We used a range of popular LLMs to identify indicators of adolescent (ages 15-17 years) depression symptoms in mental health notes. Within the evaluated configurations, FLAN-T5 demonstrated the most favorable balance between precision and recall. These results support the feasibility of using off-the-shelf LLMs for symptom detection in adolescent clinical notes, while future work using standardized prompting strategies and task formulations will help establish more definitive comparisons between model architectures. The computational efficiency of FLAN-T5 further supports its potential for deployment in resource-limited clinical settings. Our findings require further validation for broader populations and other mental health conditions.