As generative AI models such as large language models (LLMs) become more pervasive, ensuring the safety, robustness, and overall trustworthiness of these systems is paramount. However, AI is currently facing a reproducibility crisis driven by unreliable evaluations and unrepeatable experimental results. While human raters are often used to assess models for utility and safety, they introduce divergent biases and subjective opinions into their annotations. Overcoming this variance is exceptionally challenging because very little data exists to study how experimental repeatability actually improves as the annotator pool grows. Standard evaluation practices typically rely on a small number of annotations per item (often 3 to 5) and lack the persistent rater identifiers necessary to model individual variance across items. In this work, we introduce a multi-level bootstrapping approach to realistically model annotator behavior. Leveraging datasets with a large number of ratings and persistent rater identifiers, we analyze the tradeoffs between the number of items ($N$) and the number of responses per item ($K$) required to achieve statistical significance.

翻译：随着大语言模型等生成式AI模型的日益普及，确保这些系统的安全性、鲁棒性和整体可信度至关重要。然而，当前人工智能领域正面临由不可靠评估和不可重复实验结果驱动的可重复性危机。尽管人类评估者常被用于评定模型的实用性和安全性，但他们在标注中会引入不同的偏见和主观意见。克服这种变异性极具挑战性，因为目前几乎没有数据能够研究标注者池扩大时实验可重复性究竟如何提升。标准的评估实践通常依赖每项任务少量标注（通常3至5条），且缺乏必要的持续评估者标识符来建模个体在各项任务间的变异性。在本工作中，我们提出一种多层次自助法来真实模拟标注者行为。通过利用包含大量评分和持续评估者标识符的数据集，我们分析了达成统计显著性所需的项目数量（$N$）与每项任务响应数量（$K$）之间的权衡关系。