We present a human-in-the-loop evaluation framework for fact-checking novel misinformation claims and identifying social media messages that support them. Our approach extracts check-worthy claims, which are aggregated and ranked for review. Stance classifiers are then used to identify tweets supporting novel misinformation claims, which are further reviewed to determine whether they violate relevant policies. To demonstrate the feasibility of our approach, we develop a baseline system based on modern NLP methods for human-in-the-loop fact-checking in the domain of COVID-19 treatments. We make our data and detailed annotation guidelines available to support the evaluation of human-in-the-loop systems that identify novel misinformation directly from raw user-generated content.

翻译：我们提出了一种面向事实核查新型错误信息主张并识别支持性社交媒体信息的人机协同评估框架。该方法通过提取值得核查的主张，经聚合与排序后提交人工审阅。随后利用立场分类器识别支持新型错误信息主张的推文，并进一步审阅以确定其是否违反相关政策。为验证该方法的可行性，我们基于现代自然语言处理方法构建了一个基线系统，用于COVID-19治疗领域的人机协同事实核查。我们公开了数据集与详细标注指南，以支持从原始用户生成内容中直接识别新型错误信息的人机协同系统评估。