The Collaborative Qualitative Analysis (CQA) process can be time-consuming and resource-intensive, requiring multiple discussions among team members to refine codes and ideas before reaching a consensus. To address these challenges, we introduce CollabCoder, a system leveraging Large Language Models (LLMs) to support three CQA stages: independent open coding, iterative discussions, and the development of a final codebook. In the independent open coding phase, CollabCoder provides AI-generated code suggestions on demand, and allows users to record coding decision-making information (e.g. keywords and certainty) as support for the process. During the discussion phase, CollabCoder helps to build mutual understanding and productive discussion by sharing coding decision-making information with the team. It also helps to quickly identify agreements and disagreements through quantitative metrics, in order to build a final consensus. During the code grouping phase, CollabCoder employs a top-down approach for primary code group recommendations, reducing the cognitive burden of generating the final codebook. An evaluation involving 16 users confirmed the usability and effectiveness of CollabCoder and offered empirical insights into the LLMs' roles in CQA.
翻译:协作式定性分析(CQA)过程往往耗时且资源密集,需要团队成员进行多次讨论以完善编码和想法,最终达成共识。为应对这些挑战,我们提出了CollabCoder系统,该系统利用大语言模型(LLMs)支持CQA的三个阶段:独立开放式编码、迭代讨论以及最终编码手册的制定。在独立开放式编码阶段,CollabCoder按需提供AI生成的编码建议,并允许用户记录编码决策信息(例如关键词和确定性),以支持这一过程。在讨论阶段,CollabCoder通过向团队共享编码决策信息,帮助建立相互理解并促进富有成效的讨论。同时,它通过定量指标快速识别一致与分歧之处,以便最终达成共识。在编码分组阶段,CollabCoder采用自上而下的方法提供初级编码组推荐,从而减轻生成最终编码手册时的认知负担。一项包含16名用户的评估验证了CollabCoder的可用性和有效性,并提供了关于LLMs在CQA中所扮演角色的实证见解。