NLP-assisted solutions have gained considerable traction to support qualitative data analysis. However, no unified evaluation framework exists which can account for the many different settings in which qualitative researchers may employ them. In this paper, we propose an evaluation framework to study the way collaboration settings may produce different outcomes across a variety of interactive systems. Specifically, we study the impact of synchronous vs. asynchronous collaboration using three different NLP-assisted qualitative research tools and present a comprehensive analysis of significant differences in the consistency, cohesiveness, and correctness of their outputs.

翻译：自然语言处理辅助解决方案在支持定性数据分析方面已获得显著关注。然而，目前尚不存在能够涵盖定性研究人员可能采用多种不同应用场景的统一评估框架。本文提出一种评估框架，用于研究协作模式如何在各类交互式系统中产生不同的结果。具体而言，我们使用三种不同的自然语言处理辅助定性研究工具，研究了同步协作与异步协作的影响，并对其输出结果在一致性、凝聚性和正确性方面的显著差异进行了全面分析。