Hateful content online is often expressed using fact-like, not necessarily correct information, especially in coordinated online harassment campaigns and extremist propaganda. Failing to jointly address hate speech (HS) and misinformation can deepen prejudice, reinforce harmful stereotypes, and expose bystanders to psychological distress, while polluting public debate. Moreover, these messages require more effort from content moderators because they must assess both harmfulness and veracity, i.e., fact-check them. To address this challenge, we release WSF-ARG+, the first dataset which combines hate speech with check-worthiness information. We also introduce a novel LLM-in-the-loop framework to facilitate the annotation of check-worthy claims. We run our framework, testing it with 12 open-weight LLMs of different sizes and architectures. We validate it through extensive human evaluation, and show that our LLM-in-the-loop framework reduces human effort without compromising the annotation quality of the data. Finally, we show that HS messages with check-worthy claims show significantly higher harassment and hate, and that incorporating check-worthiness labels improves LLM-based HS detection up to 0.213 macro-F1 and to 0.154 macro-F1 on average for large models.

翻译：网络上的仇恨内容常以看似事实但未必正确的信息形式呈现，尤其在协调性网络骚扰运动和极端主义宣传中尤为突出。若未能同时处理仇恨言论与错误信息，将加深偏见、强化有害刻板印象，使旁观者遭受心理困扰，同时污染公共讨论空间。此外，这类信息需要内容审核员投入更多精力，因为他们必须同时评估其危害性与真实性，即进行事实核查。为应对这一挑战，我们发布了WSF-ARG+数据集——首个将仇恨言论与可核查性信息相结合的数据集。同时提出一种新型大型语言模型在回环框架，用于辅助标注需核查的主张。我们使用12种不同规模与架构的开源权重LLM对该框架进行测试，并通过广泛人工评估验证其效果。结果表明，该LLM在回环框架能在不降低数据标注质量的前提下减少人工投入。最后，我们发现含有可核查主张的仇恨言论消息表现出显著更高的骚扰与仇恨强度，而引入可核查性标签可使基于LLM的仇恨言论检测性能提升至多0.213宏F1值，大型模型平均提升0.154宏F1值。