Recommendation systems increasingly depend on massive human-labeled datasets; however, the human annotators hired to generate these labels increasingly come from homogeneous backgrounds. This poses an issue when downstream predictive models -- based on these labels -- are applied globally to a heterogeneous set of users. We study this disconnect with respect to the labels themselves, asking whether they are ``consistently conceptualized'' across annotators of different demographics. In a case study of video game labels, we conduct a survey on 5,174 gamers, identify a subset of inconsistently conceptualized game labels, perform causal analyses, and suggest both cultural and linguistic reasons for cross-country differences in label annotation. We further demonstrate that predictive models of game annotations perform better on global train sets as opposed to homogeneous (single-country) train sets. Finally, we provide a generalizable framework for practitioners to audit their own data annotation processes for consistent label conceptualization, and encourage practitioners to consider global inclusivity in recommendation systems starting from the early stages of annotator recruitment and data-labeling.

翻译：推荐系统日益依赖大规模人工标注数据集；然而，负责生成这些标签的人工标注者背景日趋同质化。当基于这些标签的下游预测模型被全球范围内异质性用户群体应用时，便会产生问题。我们针对标签本身研究这种脱节现象，探究不同人口统计学特征的标注者是否对标签存在"一致的概念化理解"。在电子游戏标签的案例研究中，我们对5,174名游戏玩家开展问卷调查，识别出概念化理解不一致的游戏标签子集，进行因果分析，并揭示国家间标签标注差异的文化与语言学原因。我们进一步证明，相比同质化（单国别）训练集，游戏标注预测模型在全球化训练集上表现更优。最后，我们为从业者提供可泛化的审计框架，用于检测其自身数据标注流程中标签概念化的一致性，并鼓励从业者从标注者招募与数据标注的早期阶段即考虑推荐系统的全球包容性。