We present a human-in-the-loop evaluation framework for fact-checking novel misinformation claims and identifying social media messages that support them. Our approach extracts check-worthy claims, which are aggregated and ranked for review. Stance classifiers are then used to identify tweets supporting novel misinformation claims, which are further reviewed to determine whether they violate relevant policies. To demonstrate the feasibility of our approach, we develop a baseline system based on modern NLP methods for human-in-the-loop fact-checking in the domain of COVID-19 treatments. We make our data and detailed annotation guidelines available to support the evaluation of human-in-the-loop systems that identify novel misinformation directly from raw user-generated content.

翻译：我们提出了一种用于事实核查新型错误信息主张并识别支持这些主张的社交媒体消息的人机协同评估框架。该框架提取需核验的主张，经聚合与排序后供人工审核。随后利用立场分类器识别支持新型错误信息主张的推文，再进一步审核以判定其是否违反相关政策。为验证该方法的可行性，我们基于现代自然语言处理方法，在COVID-19治疗领域构建了一套人机协同事实核查基线系统。我们公开了相关数据集与详细标注指南，以支撑对直接从原始用户生成内容中识别新型错误信息的人机协同系统的评估工作。