The pervasive spread of misinformation and disinformation poses a significant threat to society. Professional fact-checkers play a key role in addressing this threat, but the vast scale of the problem forces them to prioritize their limited resources. This prioritization may consider a range of factors, such as varying risks of harm posed to specific groups of people. In this work, we investigate potential implications of using a large language model (LLM) to facilitate such prioritization. Because fact-checking impacts a wide range of diverse segments of society, it is important that diverse views are represented in the claim prioritization process. This paper examines whether a LLM can reflect the views of various groups when assessing the harms of misinformation, focusing on gender as a primary variable. We pose two central questions: (1) To what extent do prompts with explicit gender references reflect gender differences in opinion in the United States on topics of social relevance? and (2) To what extent do gender-neutral prompts align with gendered viewpoints on those topics? To analyze these questions, we present the TopicMisinfo dataset, containing 160 fact-checked claims from diverse topics, supplemented by nearly 1600 human annotations with subjective perceptions and annotator demographics. Analyzing responses to gender-specific and neutral prompts, we find that GPT 3.5-Turbo reflects empirically observed gender differences in opinion but amplifies the extent of these differences. These findings illuminate AI's complex role in moderating online communication, with implications for fact-checkers, algorithm designers, and the use of crowd-workers as annotators. We also release the TopicMisinfo dataset to support continuing research in the community.

翻译：虚假信息和错误信息的普遍传播对社会构成重大威胁。专业的事实核查人员在应对这一威胁中扮演关键角色，但问题的巨大规模迫使他们将有限的资源优先分配。这种优先分配可能考虑一系列因素，例如针对特定人群的不同危害风险。在本研究中，我们探讨使用大语言模型（LLM）促进此类优先分配的潜在影响。由于事实核查影响社会中广泛多样的群体，因此在主张优先排序过程中反映多元观点至关重要。本文考察LLM在评估虚假信息危害时能否反映不同群体的观点，重点关注性别作为主要变量。我们提出两个核心问题：（1）带有明确性别指代的提示在多大程度上反映了美国社会中与社交议题相关的性别观点差异？（2）性别中立的提示在多大程度上与这些议题上的性别视角一致？为分析这些问题，我们提出了TopicMisinfo数据集，包含来自不同主题的160个经过事实核查的主张，附有近1600条包含主观认知和注释者人口统计信息的人工标注。通过分析针对特定性别和中性提示的响应，我们发现GPT 3.5-Turbo能反映经验观察到的性别观点差异，但夸大了这些差异的程度。这些发现揭示了人工智能在调节在线交流中的复杂作用，对事实核查人员、算法设计人员以及使用众包工作者作为注释者具有启示意义。我们还发布了TopicMisinfo数据集，以支持社区内的持续研究。