Misinformation is a complex societal issue, and mitigating solutions are difficult to create due to data deficiencies. To address this, we have curated the largest collection of (mis)information datasets in the literature, totaling 75. From these, we evaluated the quality of 36 datasets that consist of statements or claims, as well as the 9 datasets that consist of data in purely paragraph form. We assess these datasets to identify those with solid foundations for empirical work and those with flaws that could result in misleading and non-generalizable results, such as spurious correlations, or examples that are ambiguous or otherwise impossible to assess for veracity. We find the latter issue is particularly severe and affects most datasets in the literature. We further provide state-of-the-art baselines on all these datasets, but show that regardless of label quality, categorical labels may no longer give an accurate evaluation of detection model performance. Finally, we we propose and highlight Evaluation Quality Assessment (EQA) as a tool to guide the field toward systemic solutions rather than inadvertently propagating issues in evaluation. Overall, this guide aims to provide a roadmap for higher quality data and better grounded evaluations, ultimately improving research in misinformation detection. All datasets and other artifacts are available at misinfo-datasets.complexdatalab.com.

翻译：虚假信息是一个复杂的社会问题，由于数据缺陷，缓解方案的制定十分困难。为此，我们整理了文献中规模最大的（虚假）信息数据集集合，总计75个。在此基础上，我们评估了其中36个由陈述或主张组成的数据集的质量，以及9个完全由段落形式数据组成的数据集。我们评估这些数据集，旨在识别哪些具有坚实的实证工作基础，哪些存在可能导致误导性和不可泛化结果的缺陷，例如伪相关性，或那些模糊不清、无法评估其真实性的示例。我们发现后一个问题尤为严重，影响了文献中的大多数数据集。我们进一步为所有这些数据集提供了最先进的基线模型，但结果表明，无论标签质量如何，分类标签可能已无法准确评估检测模型的性能。最后，我们提出并强调评估质量评估（EQA）作为一种工具，以引导该领域走向系统性解决方案，而非无意中传播评估中的问题。总体而言，本指南旨在为更高质量的数据和更扎实的评估提供路线图，最终改进虚假信息检测的研究。所有数据集及其他相关资源可在 misinfo-datasets.complexdatalab.com 获取。