Large language models (LLMs) are increasingly being integrated into research workflows. However, LLMs have been shown to struggle with difficult and nuanced concepts such as those found in computational social science (CSS) research. Within the CSS community, there has been a call for new systems to be developed which center humans in LLM-supported scientific workflows. We develop AnnotateThis, a human-centered system for inspecting and improving LLM annotations, a process we refer to as LLM grounding for a target concept. AnnotateThis is developed with both computational and social scientists to reflect existing workflows for data annotation. It includes a range of information features for users to interrogate the quality and reliability of LLM annotations. We evaluate our system in two settings. In the first, we assume a researcher may not have access to ground truth data and that users of AnnotateThis have limited prior knowledge of the concept they would like an LLM to annotate. That is, they may be conducting concept specification and LLM grounding simultaneously. In the second setting, we assume access to ground truth labels and that the concept is specified for a given annotation task; here, the task of LLM grounding is more straightforward. We find that in both settings users can improve the quality of LLM annotations with AnnotateThis and that their final annotations far surpass those created without human intervention. For example, when we evaluate with ground truth labels, we see an absolute improvement of 0.15 in F-Measure and 0.23 in accuracy over a fully automated state-of-the-art method for prompt refinement.

翻译：大型语言模型（LLM）正日益融入研究工作流程。然而，研究表明LLM在处理计算社会科学（CSS）研究中复杂且微妙的概念时存在困难。CSS领域已呼吁开发以人为中心的LLM辅助科学工作流系统。我们提出AnnotateThis——一个以人为中心、用于检验与改进LLM标注的系统，这一过程称为针对目标概念的LLM接地（LLM grounding）。该系统由计算科学家与社会科学家共同开发，以反映现有数据标注工作流程，并包含多维度信息特征，供用户检验LLM标注的质量与可靠性。我们在两种场景下评估该系统：场景一假定研究者可能无法获取真实标签（ground truth），且AnnotateThis用户对LLM需标注的概念仅有有限先验知识——即用户可能同时进行概念界定与LLM接地；场景二假定用户可获取真实标签，且概念已针对特定标注任务明确界定——此时LLM接地的任务更为直接。实验发现，在两种场景中，用户均可借助AnnotateThis提升LLM标注质量，其最终标注结果显著优于无人工干预的标注。例如，在基于真实标签的评估中，相较于全自动最先进的提示优化方法，我们在F值上实现0.15的绝对提升，准确率上实现0.23的绝对提升。