Conversational question-answering (CQA) systems aim to create interactive search systems that effectively retrieve information by interacting with users. To replicate human-to-human conversations, existing work uses human annotators to play the roles of the questioner (student) and the answerer (teacher). Despite its effectiveness, challenges exist as human annotation is time-consuming, inconsistent, and not scalable. To address this issue and investigate the applicability of large language models (LLMs) in CQA simulation, we propose a simulation framework that employs zero-shot learner LLMs for simulating teacher-student interactions. Our framework involves two LLMs interacting on a specific topic, with the first LLM acting as a student, generating questions to explore a given search topic. The second LLM plays the role of a teacher by answering questions and is equipped with additional information, including a text on the given topic. We implement both the student and teacher by zero-shot prompting the GPT-4 model. To assess the effectiveness of LLMs in simulating CQA interactions and understand the disparities between LLM- and human-generated conversations, we evaluate the simulated data from various perspectives. We begin by evaluating the teacher's performance through both automatic and human assessment. Next, we evaluate the performance of the student, analyzing and comparing the disparities between questions generated by the LLM and those generated by humans. Furthermore, we conduct extensive analyses to thoroughly examine the LLM performance by benchmarking state-of-the-art reading comprehension models on both datasets. Our results reveal that the teacher LLM generates lengthier answers that tend to be more accurate and complete. The student LLM generates more diverse questions, covering more aspects of a given topic.

翻译：对话式问答系统旨在创建交互式搜索系统，通过与用户互动有效检索信息。为复现人人对话场景，现有研究使用人类标注者扮演提问者（学生）与回答者（教师）角色。尽管该方法有效，但存在人工标注耗时、不一致且不可扩展等问题。为解决这一难题并探究大语言模型（LLMs）在对话式问答模拟中的适用性，我们提出了一种模拟框架，采用零样本学习型LLMs模拟师生互动。该框架包含两个针对特定主题交互的LLM：首个LLM作为学生，生成问题以探索既定搜索主题；第二个LLM扮演教师角色，在回答问题时配备包含指定主题文本的额外信息。我们通过零样本提示GPT-4模型实现师生双方。为评估LLMs模拟对话式问答交互的有效性并理解LLM生成对话与人类生成对话的差异，我们从多个维度评估模拟数据：首先通过自动评估与人工评估检验教师表现；其次评估学生表现，分析比较LLM生成问题与人类生成问题之间的差异；此外，我们通过在两数据集上基准测试最先进的阅读理解模型，开展广泛分析以全面检验LLM性能。实验结果表明：教师LLM能生成更准确完整的长答案，学生LLM则能产生覆盖主题更多维度的多样化问题。