Large language models (LLMs) have demonstrated their capabilities across various NLP tasks. Their potential in e-commerce is also substantial, evidenced by practical implementations such as platform search, personalized recommendations, and customer service. One primary concern associated with LLMs is their factuality (e.g., hallucination), which is urgent in e-commerce due to its significant impact on user experience and revenue. Despite some methods proposed to evaluate LLMs' factuality, issues such as lack of reliability, high consumption, and lack of domain expertise leave a gap between effective assessment in e-commerce. To bridge the evaluation gap, we propose ECKGBench, a dataset specifically designed to evaluate the capacities of LLMs in e-commerce knowledge. Specifically, we adopt a standardized workflow to automatically generate questions based on a large-scale knowledge graph, guaranteeing sufficient reliability. We employ the simple question-answering paradigm, substantially improving the evaluation efficiency by the least input and output tokens. Furthermore, we inject abundant e-commerce expertise in each evaluation stage, including human annotation, prompt design, negative sampling, and verification. Besides, we explore the LLMs' knowledge boundaries in e-commerce from a novel perspective. Through comprehensive evaluations of several advanced LLMs on ECKGBench, we provide meticulous analysis and insights into leveraging LLMs for e-commerce.

翻译：大语言模型（LLMs）已在各类自然语言处理任务中展现出卓越能力。其在电子商务领域的潜力同样巨大，平台搜索、个性化推荐和客户服务等实际应用已证实这一点。大语言模型面临的核心问题之一是其事实准确性（例如幻觉现象），这在电子商务领域尤为紧迫，因其直接影响用户体验与营收。尽管已有部分评估大语言模型事实性的方法，但存在可靠性不足、消耗成本高、缺乏领域专业知识等问题，导致针对电子商务场景的有效评估仍存空白。为填补这一评估缺口，我们提出ECKGBench——一个专门用于评估大语言模型在电子商务知识领域能力的基准数据集。具体而言，我们采用标准化工作流程，基于大规模知识图谱自动生成问题，确保评估的充分可靠性。通过采用简洁的问答范式，以最少的输入输出标记显著提升评估效率。此外，我们在每个评估阶段（包括人工标注、提示设计、负样本采样及验证环节）注入丰富的电子商务领域专业知识。同时，我们以创新视角探索大语言模型在电子商务领域的知识边界。通过对多个先进大语言模型在ECKGBench上的综合评估，我们为电子商务场景下大语言模型的应用提供了细致的分析与洞见。