Q-Eval-100K: Evaluating Visual Quality and Alignment Level for Text-to-Vision Content
Journal:
arXiv
Published Date:
Mar 4, 2025
Abstract
Evaluating text-to-vision content hinges on two crucial aspects: visual
quality and alignment. While significant progress has been made in developing
objective models to assess these dimensions, the performance of such models
heavily relies on the scale and quality of human annotations. According to
Scaling Law, increasing the number of human-labeled instances follows a
predictable pattern that enhances the performance of evaluation models.
Therefore, we introduce a comprehensive dataset designed to Evaluate Visual
quality and Alignment Level for text-to-vision content (Q-EVAL-100K), featuring
the largest collection of human-labeled Mean Opinion Scores (MOS) for the
mentioned two aspects. The Q-EVAL-100K dataset encompasses both text-to-image
and text-to-video models, with 960K human annotations specifically focused on
visual quality and alignment for 100K instances (60K images and 40K videos).
Leveraging this dataset with context prompt, we propose Q-Eval-Score, a unified
model capable of evaluating both visual quality and alignment with special
improvements for handling long-text prompt alignment. Experimental results
indicate that the proposed Q-Eval-Score achieves superior performance on both
visual quality and alignment, with strong generalization capabilities across
other benchmarks. These findings highlight the significant value of the
Q-EVAL-100K dataset. Data and codes will be available at
https://github.com/zzc-1998/Q-Eval.
