Large language models (LLMs), such as ChatGPT and GPT-4, are gaining wide-spread real world use. Yet, these LLMs are closed source, and little is known about their performance in real-world use cases. In this paper, we apply and evaluate the combination of ChatGPT and GPT-4 for the real-world task of mining insights from a text corpus in order to identify research challenges in the field of HCI. We extract 4,392 research challenges in over 100 topics from the 2023~CHI conference proceedings and visualize the research challenges for interactive exploration. We critically evaluate the LLMs on this practical task and conclude that the combination of ChatGPT and GPT-4 makes an excellent cost-efficient means for analyzing a text corpus at scale. Cost-efficiency is key for flexibly prototyping research ideas and analyzing text corpora from different perspectives, with implications for applying LLMs for mining insights in academia and practice.

翻译：以ChatGPT和GPT-4为代表的大型语言模型（LLMs）正在现实世界中获得广泛应用。然而，这些LLMs属于闭源系统，其在实际应用场景中的性能表现尚不明确。本文针对从文本语料库中挖掘洞察以识别人机交互（HCI）领域研究挑战这一现实任务，应用并评估了ChatGPT与GPT-4的组合方案。我们从2023年CHI会议论文集中提取了涵盖100多个主题的4,392个研究挑战，并将其可视化以供交互式探索。我们对此实际任务中的LLMs性能进行了批判性评估，结论表明：ChatGPT与GPT-4的组合为大规模文本语料分析提供了一种卓越的高性价比手段。成本效益对于灵活原型化研究思路以及从多视角分析文本语料至关重要，这对在学术界与实践领域应用LLMs进行洞察挖掘具有重要启示。