Evaluating Large Multimodal Models for Nutrition Analysis: A Benchmark Enriched with Contextual Metadata
Journal:
arXiv
Published Date:
Jul 9, 2025
Abstract
Large Multimodal Models (LMMs) are increasingly applied to meal images for
nutrition analysis. However, existing work primarily evaluates proprietary
models, such as GPT-4. This leaves the broad range of LLMs underexplored.
Additionally, the influence of integrating contextual metadata and its
interaction with various reasoning modifiers remains largely uncharted. This
work investigates how interpreting contextual metadata derived from GPS
coordinates (converted to location/venue type), timestamps (transformed into
meal/day type), and the food items present can enhance LMM performance in
estimating key nutritional values. These values include calories,
macronutrients (protein, carbohydrates, fat), and portion sizes. We also
introduce ACETADA, a new food-image dataset slated for public release. This
open dataset provides nutrition information verified by the dietitian and
serves as the foundation for our analysis. Our evaluation across eight LMMs
(four open-weight and four closed-weight) first establishes the benefit of
contextual metadata integration over straightforward prompting with images
alone. We then demonstrate how this incorporation of contextual information
enhances the efficacy of reasoning modifiers, such as Chain-of-Thought,
Multimodal Chain-of-Thought, Scale Hint, Few-Shot, and Expert Persona.
Empirical results show that integrating metadata intelligently, when applied
through straightforward prompting strategies, can significantly reduce the Mean
Absolute Error (MAE) and Mean Absolute Percentage Error (MAPE) in predicted
nutritional values. This work highlights the potential of context-aware LMMs
for improved nutrition analysis.
