Generative Artificial Intelligence (GenAI) is now widespread in education, yet the efficacy of GenAI systems remains constrained by the quality and interpretation of the labeled data used to train and evaluate them. Studies commonly report inter-rater reliability (IRR), often summarized by a single coefficient such as Cohen's kappa (k), as a gatekeeper to ``ground truth.'' We argue that many educational assessment and practice support settings include challenges, such as high-inference constructs, skewed label distributions, and temporally segmented multimodal data, which yield potential misapplication or misinterpretation of threshold-based heuristics for IRR. The growing use of large language models as annotators and judges introduces risks such as automation bias and circular validation. We propose four practical shifts for establishing ground truth: (1) treat IRR as a diagnostic signal to localize disagreement and refine constructs rather than a mechanical acceptance threshold (e.g., k > 0.8); (2) require transparent reporting of rater expertise, codebook development, reconciliation procedures, and segmentation rules; (3) mitigate risks in LLM annotation through bias audits and verification workflows; and (4) complement agreement statistics with validity and effectiveness evidence for the intended use, including uncertainty-aware labeling (e.g., assigning different labels to the same item to capture nuance), criterion-related checks (e.g., predictive tests to check if labels forecast the intended outcome), and close-the-loop evaluations of whether systems trained on these labels improve learning beyond a reasonable control. We illustrate these shifts through case studies of multimodal tutoring data and provide actionable recommendations toward strengthening the evidence base of labeled AIED datasets.

翻译：摘要：生成式人工智能（GenAI）现已广泛应用于教育领域，然而，GenAI系统的效能仍受制于用于训练和评估的标注数据的质量及其解读方式。研究通常将评分者间信度（Inter-Rater Reliability, IRR），通常以单一系数（如Cohen's Kappa，k）概而言之，作为“地面真值”的鉴定标准。我们认为，许多教育评估与实践支持场景存在高推理性构念、偏斜标签分布以及时间分段多模态数据等挑战，这可能导致基于阈值的IRR启发式方法的误用或误读。大型语言模型作为标注者和评判者的日益普及，引入了自动偏差和循环验证等风险。我们提出建立地面真值的四项实用转变：(1) 将IRR视为定位分歧并完善构念的诊断信号，而非机械的接受阈值（例如k>0.8）；(2) 要求透明报告标注者专长、编码手册开发、调和流程及分割规则；(3) 通过偏差审计和验证工作流程降低LLM标注中的风险；(4) 用针对预期用途的有效性和效度证据来补充一致性统计，包括不确定性感知标注（例如为同一项目分配不同标签以捕捉细微差异）、基于标准的检验（例如预测性测试以检查标签是否预测预期结果），以及闭环评估：检验基于这些标签训练的系统是否能超越合理基线提升学习效果。我们通过多模态辅导数据的案例研究阐释这些转变，并提供可操作建议，以加强标注AIED数据集的证据基础。