Image-Based Dietary Assessment Using Multimodal Large Language Models: Performance, Limitations, and Clinical Applicability in Diabetes
Journal:
medRxiv
Published Date:
Oct 24, 2026
(1)
Abstract
Background Image-based dietary assessment is useful for precision diabetes care. Multimodal Large Language Models (mLLMs) may greatly facilitate dietary self-monitoring. However, comprehensive evaluation of their real-world accuracy remains limited. Methods We developed SeeFood, an open-source benchmark, and evaluated eight commercial and open-source mLLMs on 80 real-world meal photographs from the SNAPMe dataset for nutrient and energy estimation (calories, carbohydrates, protein, fat, fiber), food-item identification, and day-level diet quality (Healthy Eating Index, HEI-2020). Three finalists were evaluated on the complete dataset (1680 photographs, 275 participant-days), and the best model was analyzed by meal type, portion size, sizing-marker use, ingredient visibility and meal complexity. Results GPT-4o mini achieved the lowest errors, with a significant advantage for carbohydrates; the three finalists performed similarly on other outcomes and on HEI-2020. Mean absolute percentage errors (MAPE) remained high (52%--90%), and the direction of bias was model-specific. The carbohydrate mean absolute error was 13.3 g (95% CI 12.4--14.3), and only 57% of meals were within +/- 10 g, a clinically relevant threshold for insulin dosing in type 1 diabetes. Calorie errors for lunch and dinner were in the range reported for human estimation on other images (~55% MAPE), whereas drink-only images had the highest relative errors (312% MAPE) because of dissolved sugars. Food items were reliably identified when fully visible (Dice = 0.74), and HEI-2020 estimates showed moderate agreement with reference values (RMSE = 12.5 points; r = 0.61). Larger portions were increasingly underestimated, and sizing references did not improve accuracy. Newer or larger models and chain-of-thought prompting did not consistently improve performance. Conclusions Current mLLMs can extract meaningful dietary information from real-world meal photographs, but substantial errors in nutrient and portion-size estimation currently limit their use for precision diabetes management. Integrating visual assessment with additional contextual information may enable more reliable, lower-burden dietary monitoring.
