In recent years, social media companies have grappled with defining and enforcing content moderation policies surrounding political content on their platforms, due in part to concerns about political bias, disinformation, and polarization. These policies have taken many forms, including disallowing political advertising, limiting the reach of political topics, fact-checking political claims, and enabling users to hide political content altogether. However, implementing these policies requires human judgement to label political content, and it is unclear how well human labelers perform at this task, or whether biases affect this process. Therefore, in this study we experimentally evaluate the feasibility and practicality of using crowd workers to identify political content, and we uncover biases that make it difficult to identify this content. Our results problematize crowds composed of seemingly interchangeable workers, and provide preliminary evidence that aggregating judgements from heterogeneous workers may help mitigate political biases. In light of these findings, we identify strategies to achieving fairer labeling outcomes, while also better supporting crowd workers at this task and potentially mitigating biases.

翻译：近年来，社交媒体公司在其平台上围绕政治内容定义和执行内容审核政策时面临重重困难，部分原因在于对政治偏见、虚假信息和社会极化的担忧。这些政策采取了多种形式，包括禁止政治广告、限制政治话题传播范围、核查政治主张的真实性，以及允许用户完全隐藏政治内容。然而，实施这些政策需要依赖人类判断来标注政治内容，但目前尚不清楚人类标注者在此任务中的表现如何，也不确定偏见是否会影响这一过程。因此，本研究通过实验评估了使用众包工作者识别政治内容的可行性与实用性，并揭示了导致该内容难以识别的偏见。我们的研究结果对由看似可互换的工作者组成的众包群体提出了质疑，并提供了初步证据表明，汇总来自异质性工作者的判断可能有助于缓解政治偏见。基于这些发现，我们提出了实现更公平标注结果的策略，同时更好地支持众包工作者完成此任务，并可能减轻偏见的影响。