For machine learning datasets to accurately represent diverse opinions in a population, they must preserve variation in data labels while filtering out spam or low-quality responses. How can we balance annotator reliability and representation? We empirically evaluate how a range of heuristics for annotator filtering affect the preservation of variation on subjective tasks. We find that these methods, designed for contexts in which variation from a single ground-truth label is considered noise, often remove annotators who disagree instead of spam annotators, introducing suboptimal tradeoffs between accuracy and label diversity. We find that conservative settings for annotator removal (<5%) are best, after which all tested methods increase the mean absolute error from the true average label. We analyze performance on synthetic spam to observe that these methods often assume spam annotators are more random than real spammers tend to be: most spammers are distributionally indistinguishable from real annotators, and the minority that are distinguishable tend to give relatively fixed answers, not random ones. Thus, tasks requiring the preservation of variation reverse the intuition of existing spam filtering methods: spammers tend to be less random than non-spammers, so metrics that assume variation is spam fare worse. These results highlight the need for spam removal methods that account for label diversity.

翻译：为使机器学习数据集准确反映群体中的多元观点，必须在过滤垃圾信息或低质量响应的同时保留数据标签的多样性。我们应如何平衡标注者可靠性与代表性？本文通过实证评估一系列标注者过滤启发式方法对主观任务中多样性保持的影响。研究发现，这些原本为单一标准答案场景下将标签差异视为噪声而设计的方法，往往误删持不同意见的标注者而非真正的垃圾信息标注者，导致在准确性与标签多样性之间产生次优权衡。实验表明，采用保守的标注者剔除阈值（<5%）效果最佳，超过该阈值后所有测试方法均会增大与真实平均标签之间的平均绝对误差。通过对合成垃圾信息的分析发现，这些方法常假定垃圾信息标注者比实际更随机：多数垃圾信息标注者在分布特征上与真实标注者无法区分，而少数可区分的垃圾信息标注者往往给出相对固定的答案而非随机答案。因此，需要保持多样性的任务颠覆了现有垃圾信息过滤方法的直觉：垃圾信息标注者往往比非垃圾信息标注者随机性更低，故将多样性视为垃圾信息的度量指标表现更差。这些结果凸显了开发兼顾标签多样性的垃圾信息剔除方法的必要性。