The RVL-CDIP benchmark is widely used for measuring performance on the task of document classification. Despite its widespread use, we reveal several undesirable characteristics of the RVL-CDIP benchmark. These include (1) substantial amounts of label noise, which we estimate to be 8.1% (ranging between 1.6% to 16.9% per document category); (2) presence of many ambiguous or multi-label documents; (3) a large overlap between test and train splits, which can inflate model performance metrics; and (4) presence of sensitive personally-identifiable information like US Social Security numbers (SSNs). We argue that there is a risk in using RVL-CDIP for benchmarking document classifiers, as its limited scope, presence of errors (state-of-the-art models now achieve accuracy error rates that are within our estimated label error rate), and lack of diversity make it less than ideal for benchmarking. We further advocate for the creation of a new document classification benchmark, and provide recommendations for what characteristics such a resource should include.

翻译：RVL-CDIP基准测试集被广泛用于衡量文档分类任务的表现。尽管其应用广泛，但我们揭示了该基准测试集若干不良特性，包括：(1) 存在大量标注噪声，估算噪声率达8.1%（各文档类别噪声率介于1.6%至16.9%之间）；(2) 存在大量模糊或多标签文档；(3) 测试集与训练集之间存在严重重叠，可能导致模型性能指标膨胀；(4) 包含敏感的个人身份信息（如美国社会安全号码）。我们认为，使用RVL-CDIP进行文档分类器基准测试存在风险——由于其范围有限、存在误差（当前最优模型的准确率误差已接近我们估算的标注误差率）且缺乏多样性，使其难以成为理想的基准测试资源。我们进一步倡导建立新的文档分类基准测试集，并就此类资源应具备的特征提出建议。