ReGraP-LLaVA: Reasoning enabled Graph-based Personalized Large Language and Vision Assistant
Journal:
arXiv
Published Date:
May 6, 2025
Abstract
Recent advances in personalized MLLMs enable effective capture of
user-specific concepts, supporting both recognition of personalized concepts
and contextual captioning. However, humans typically explore and reason over
relations among objects and individuals, transcending surface-level information
to achieve more personalized and contextual understanding. To this end,
existing methods may face three main limitations: Their training data lacks
multi-object sets in which relations among objects are learnable. Building on
the limited training data, their models overlook the relations between
different personalized concepts and fail to reason over them. Their experiments
mainly focus on a single personalized concept, where evaluations are limited to
recognition and captioning tasks. To address the limitations, we present a new
dataset named ReGraP, consisting of 120 sets of personalized knowledge. Each
set includes images, KGs, and CoT QA pairs derived from the KGs, enabling more
structured and sophisticated reasoning pathways. We propose ReGraP-LLaVA, an
MLLM trained with the corresponding KGs and CoT QA pairs, where soft and hard
graph prompting methods are designed to align KGs within the model's semantic
space. We establish the ReGraP Benchmark, which contains diverse task types:
multiple-choice, fill-in-the-blank, True/False, and descriptive questions in
both open- and closed-ended settings. The proposed benchmark is designed to
evaluate the relational reasoning and knowledge-connection capability of
personalized MLLMs. We conduct experiments on the proposed ReGraP-LLaVA and
other competitive MLLMs. Results show that the proposed model not only learns
personalized knowledge but also performs relational reasoning in responses,
achieving the SoTA performance compared with the competitive methods. All the
codes and datasets are released at: https://github.com/xyfyyds/ReGraP.
