Community Notes, the crowd-sourced misinformation governance system on X (formerly Twitter), allows users to flag misleading posts, attach contextual notes, and rate the notes' helpfulness. However, our empirical analysis of 30.8K health-related notes reveals substantial latency, with a median delay of 17.6 hours before notes receive a helpfulness status. To improve responsiveness during real-world misinformation surges, we propose CrowdNotes+, a unified LLM-based framework that augments Community Notes for faster and more reliable health misinformation governance. CrowdNotes+ integrates two modes: (1) evidence-grounded note augmentation and (2) utility-guided note automation, supported by a hierarchical three-stage evaluation of relevance, correctness, and helpfulness. We instantiate the framework with HealthNotes, a benchmark of 1.2K health notes annotated for helpfulness, and a fine-tuned helpfulness judge. Our analysis first uncovers a key loophole in current crowd-sourced governance: voters frequently conflate stylistic fluency with factual accuracy. Addressing this via our hierarchical evaluation, experiments across 15 representative LLMs demonstrate that CrowdNotes+ significantly outperforms human contributors in note correctness, helpfulness, and evidence utility.
翻译:社区注释机制作为X平台(原推特)上众包形式的虚假信息治理系统,允许用户标记误导性帖子、附加情境注释并评估注释有效性。然而,我们对3.08万条健康相关注释的实证分析揭示了显著的延迟问题——注释获得有效性状态的中位等待时间长达17.6小时。为提升真实世界虚假信息爆发期间的响应效率,我们提出CrowdNotes+这一统一的大语言模型框架,通过增强社区注释机制实现更快、更可靠的健康虚假信息治理。CrowdNotes+融合两种模式:(1)基于证据的注释增强模式与(2)效用引导的注释自动化模式,并由相关性、正确性和有效性三级分层评估体系支撑。我们通过构建含1200条健康注释(已标注有效性)的HealthNotes基准数据集及微调的有效性评判模型实现该框架。分析首先揭示了当前众包治理的关键漏洞:投票者常将文体流畅性与事实准确性混为一谈。针对此问题,通过三级评估在15个代表性大语言模型上的实验表明,CrowdNotes+在注释准确性、有效性和证据效用方面显著优于人类贡献者。