In this paper, we investigate the use of large language models (LLMs) like ChatGPT for document-grounded response generation in the context of information-seeking dialogues. For evaluation, we use the MultiDoc2Dial corpus of task-oriented dialogues in four social service domains previously used in the DialDoc 2022 Shared Task. Information-seeking dialogue turns are grounded in multiple documents providing relevant information. We generate dialogue completion responses by prompting a ChatGPT model, using two methods: Chat-Completion and LlamaIndex. ChatCompletion uses knowledge from ChatGPT model pretraining while LlamaIndex also extracts relevant information from documents. Observing that document-grounded response generation via LLMs cannot be adequately assessed by automatic evaluation metrics as they are significantly more verbose, we perform a human evaluation where annotators rate the output of the shared task winning system, the two Chat-GPT variants outputs, and human responses. While both ChatGPT variants are more likely to include information not present in the relevant segments, possibly including a presence of hallucinations, they are rated higher than both the shared task winning system and human responses.

翻译：本文研究了在信息寻求对话场景下使用大型语言模型（如ChatGPT）进行文档基础响应生成的方法。评估使用了MultiDoc2Dial语料库，该语料库包含先前在DialDoc 2022共享任务中使用的四个社会服务领域的任务导向对话。信息寻求对话回合基于多个文档提供相关信息。我们通过提示ChatGPT模型生成对话完成响应，采用了两种方法：Chat-Completion和LlamaIndex。Chat-Completion利用ChatGPT模型预训练中的知识，而LlamaIndex还从文档中提取相关信息。观察到通过大型语言模型生成的文档基础响应因过于冗长而无法通过自动评估指标充分衡量，我们进行了人工评估，由标注者对共享任务获胜系统、两种ChatGPT变体输出以及人类响应进行评分。尽管两种ChatGPT变体更可能包含相关段落中不存在的信息（可能包含幻觉现象），但它们的评分高于共享任务获胜系统和人类响应。