UniEval: Unified Holistic Evaluation for Unified Multimodal Understanding and Generation
Journal:
arXiv
Published Date:
May 15, 2025
Abstract
The emergence of unified multimodal understanding and generation models is
rapidly attracting attention because of their ability to enhance
instruction-following capabilities while minimizing model redundancy. However,
there is a lack of a unified evaluation framework for these models, which would
enable an elegant, simplified, and overall evaluation. Current models conduct
evaluations on multiple task-specific benchmarks, but there are significant
limitations, such as the lack of overall results, errors from extra evaluation
models, reliance on extensive labeled images, benchmarks that lack diversity,
and metrics with limited capacity for instruction-following evaluation. To
tackle these challenges, we introduce UniEval, the first evaluation framework
designed for unified multimodal models without extra models, images, or
annotations. This facilitates a simplified and unified evaluation process. The
UniEval framework contains a holistic benchmark, UniBench (supports both
unified and visual generation models), along with the corresponding UniScore
metric. UniBench includes 81 fine-grained tags contributing to high diversity.
Experimental results indicate that UniBench is more challenging than existing
benchmarks, and UniScore aligns closely with human evaluations, surpassing
current metrics. Moreover, we extensively evaluated SoTA unified and visual
generation models, uncovering new insights into Univeral's unique values.
