ECKGBench: Benchmarking Large Language Models in E-commerce Leveraging Knowledge Graph
Journal:
arXiv
Published Date:
Mar 20, 2025
Abstract
Large language models (LLMs) have demonstrated their capabilities across
various NLP tasks. Their potential in e-commerce is also substantial, evidenced
by practical implementations such as platform search, personalized
recommendations, and customer service. One primary concern associated with LLMs
is their factuality (e.g., hallucination), which is urgent in e-commerce due to
its significant impact on user experience and revenue. Despite some methods
proposed to evaluate LLMs' factuality, issues such as lack of reliability, high
consumption, and lack of domain expertise leave a gap between effective
assessment in e-commerce. To bridge the evaluation gap, we propose ECKGBench, a
dataset specifically designed to evaluate the capacities of LLMs in e-commerce
knowledge. Specifically, we adopt a standardized workflow to automatically
generate questions based on a large-scale knowledge graph, guaranteeing
sufficient reliability. We employ the simple question-answering paradigm,
substantially improving the evaluation efficiency by the least input and output
tokens. Furthermore, we inject abundant e-commerce expertise in each evaluation
stage, including human annotation, prompt design, negative sampling, and
verification. Besides, we explore the LLMs' knowledge boundaries in e-commerce
from a novel perspective. Through comprehensive evaluations of several advanced
LLMs on ECKGBench, we provide meticulous analysis and insights into leveraging
LLMs for e-commerce.
