Improved Image Caption Rating - Datasets, Game, and Model.

Journal: Extended abstracts on Human factors in computing systems. CHI Conference
Published Date: Apr 19, 2023

Abstract

How well a caption fits an image can be difficult to assess due to the subjective nature of caption quality. What is a caption? We investigate this problem by focusing on image-caption ratings and by generating high quality datasets from human feedback with gamification. We validate the datasets by showing a higher level of inter-rater agreement, and by using them to train custom machine learning models to predict new ratings. Our approach outperforms previous metrics - the resulting datasets are more easily learned and are of higher quality than other currently available datasets for image-caption rating.

Authors

  • Andrew Taylor Scott
    Department of Computer Science, San Francisco State University, San Francisco, CA, USA.
  • Lothar D Narins
    Department of Computer Science, San Francisco State University, San Francisco, CA, USA.
  • Anagha Kulkarni
    Department of Computer Science, San Francisco State University, San Francisco, CA, USA.
  • Mar Castanon
    Department of Computer Science, San Francisco State University, San Francisco, CA, USA.
  • Benjamin Kao
    Department of Computer Science, San Francisco State University, San Francisco, CA, USA.
  • Shasta Ihorn
    Department of Psychology, San Francisco State University, San Francisco, CA, USA.
  • Yue-Ting Siu
    Department of Special Education, San Francisco State University, San Francisco, CA, USA.
  • Ilmi Yoon
    Department of Computer Science, San Francisco State University, San Francisco, CA, USA.

Keywords

No keywords available for this article.

External Resources

Popular Topics
Recent Journals