Optical character recognition (OCR) and document understanding systems increasingly rely on large vision and vision-language models, yet evaluation remains centered on modern, Western, and institutional documents. This emphasis masks system behavior in historical and marginalized archives, where layout, typography, and material degradation shape interpretation. This study examines how OCR and document understanding systems are evaluated, with particular attention to Black historical newspapers. We review OCR and document understanding papers, as well as benchmark datasets, which are published between 2006 and 2025 using the PRISMA framework. We look into how the studies report training data, benchmark design, and evaluation metrics for vision transformer and multimodal OCR systems. During the review, we found that Black newspapers and other community-produced historical documents rarely appear in reported training data or evaluation benchmarks. Most evaluations emphasize character accuracy and task success on modern layouts. They rarely capture structural failures common in historical newspapers, including column collapse, typographic errors, and hallucinated text. To put these findings into perspective, we use previous empirical studies and archival statistics from significant Black press collections to show how evaluation gaps lead to structural invisibility and representational harm. We propose that these gaps occur due to organizational (meso) and institutional (macro) behaviors and structure, shaped by benchmark incentives and data governance decisions.

翻译：光学字符识别（OCR）和文档理解系统日益依赖大型视觉模型及视觉-语言模型，然而评估重点仍集中于现代、西方及机构性文档。这种侧重掩盖了系统在处理历史文献和被边缘化档案时的表现，在这些场景中，版面布局、排版风格和材料老化共同影响着文本的解读。本研究旨在考察OCR及文档理解系统的评估方式，并特别关注美国黑人历史报纸。我们依据PRISMA框架，系统梳理了2006年至2025年间发表的OCR与文档理解领域论文及相关基准数据集，深入分析了这些研究在报告训练数据、基准设计以及针对视觉Transformer和多模态OCR系统的评估指标方面的具体实践。在综述过程中，我们发现黑人报纸及其他由社区制作的历史文献极少出现在报告的训练数据或评估基准中。多数评估侧重于字符准确率和在现代版面中的任务成功率，却很少捕捉历史报纸中常见的结构性故障，例如列错乱、排印错误和文本幻觉。为客观审视上述发现，我们借助以往实证研究及重要黑人报刊档案的统计资料，论证了评估缺口如何导致结构性的不可见性与表征性伤害。我们认为，这些缺口的产生源于组织（中观）和制度（宏观）层面的行为与结构，并受到基准激励措施和数据治理决策的共同塑造。