Conversational Assistants (CA) are increasingly supporting human workers in knowledge management. Traditionally, CAs respond in specific ways to predefined user intents and conversation patterns. However, this rigidness does not handle the diversity of natural language well. Recent advances in natural language processing, namely Large Language Models (LLMs), enable CAs to converse in a more flexible, human-like manner, extracting relevant information from texts and capturing information from expert humans but introducing new challenges such as ``hallucinations''. To assess the potential of using LLMs for knowledge management tasks, we conducted a user study comparing an LLM-based CA to an intent-based system regarding interaction efficiency, user experience, workload, and usability. This revealed that LLM-based CAs exhibited better user experience, task completion rate, usability, and perceived performance than intent-based systems, suggesting that switching NLP techniques can be beneficial in the context of knowledge management.

翻译：对话助手（CA）正日益在知识管理中为人类工作者提供支持。传统上，CA以特定方式响应用户预定义的意图和对话模式。然而，这种刚性结构难以妥善处理自然语言的多样性。自然语言处理领域的最新进展，特别是大型语言模型（LLM），使CA能够以更灵活、类人的方式进行对话，从文本中提取相关信息并向人类专家获取知识，但同时也引入了诸如“幻觉”等新挑战。为评估LLM在知识管理任务中的应用潜力，我们开展了一项用户研究，比较了基于LLM的CA与基于意图的系统在交互效率、用户体验、工作负荷和可用性方面的表现。研究发现，基于LLM的CA在用户体验、任务完成率、可用性和感知性能方面均优于基于意图的系统，这表明在知识管理场景中转换NLP技术可能具有显著优势。