Large language models (LLMs), such as ChatGPT and GPT-4, are gaining wide-spread real world use. Yet, these LLMs are closed source, and little is known about their performance in real-world use cases. In this paper, we apply and evaluate the combination of ChatGPT and GPT-4 for the real-world task of mining insights from a text corpus in order to identify research challenges in the field of HCI. We extract 4,392 research challenges in over 100 topics from the 2023 CHI conference proceedings and visualize the research challenges for interactive exploration. We critically evaluate the LLMs on this practical task and conclude that the combination of ChatGPT and GPT-4 makes an excellent cost-efficient means for analyzing a text corpus at scale. Cost-efficiency is key for flexibly prototyping research ideas and analyzing text corpora from different perspectives, with implications for applying LLMs for mining insights in academia and practice.

翻译：大语言模型（LLMs），如ChatGPT和GPT-4，正获得广泛的现实世界应用。然而，这些LLMs是闭源的，且其在真实用例中的性能尚不明确。本文应用并评估了ChatGPT与GPT-4的组合，执行从文本语料库中挖掘洞察的现实任务，以识别人机交互领域的研究挑战。我们从2023年CHI会议论文集提取了涵盖100多个主题的4,392个研究挑战，并对其进行了可视化以支持交互式探索。针对这一实际任务，我们对LLMs进行了批判性评估，结论是：ChatGPT与GPT-4的组合在大规模分析文本语料库方面具有极佳的成本效益。成本效益对于灵活原型化研究思路以及从不同角度分析文本语料库至关重要，这对在学术界和实践中应用LLMs进行洞察挖掘具有重要启示。