Collaborative problem solving (CPS) is widely recognized as a critical 21st century skill. Efficiently coding communication data is a big challenge in scaling up research on assessing CPS. This paper reports the findings on using ChatGPT to directly code CPS chat data by benchmarking performance across multiple datasets and coding frameworks. We found that ChatGPT-based coding outperformed human coding in tasks where the discussions were characterized by colloquial languages but fell short in tasks where the discussions dealt with specialized scientific terminology and contexts. The findings offer practical guidelines for researchers to develop strategies for efficient and scalable analysis of communication data from CPS tasks.

翻译：协作问题解决（CPS）被广泛视为21世纪的关键能力。在扩大CPS评估研究规模的过程中，如何高效编码交流数据是一大挑战。本文通过在多数据集和编码框架下进行性能基准测试，报告了使用ChatGPT直接编码CPS聊天数据的研究发现。我们发现，在讨论内容以日常口语为特征的任务中，基于ChatGPT的编码表现优于人工编码；而在涉及专业科学术语与情境的讨论任务中，其表现尚有不足。这些发现为研究者制定高效、可扩展的CPS任务交流数据分析策略提供了实践指导。