Evidence data for automated fact-checking (AFC) can be in multiple modalities such as text, tables, images, audio, or video. While there is increasing interest in using images for AFC, previous works mostly focus on detecting manipulated or fake images. We propose a novel task, chart-based fact-checking, and introduce ChartBERT as the first model for AFC against chart evidence. ChartBERT leverages textual, structural and visual information of charts to determine the veracity of textual claims. For evaluation, we create ChartFC, a new dataset of 15, 886 charts. We systematically evaluate 75 different vision-language (VL) baselines and show that ChartBERT outperforms VL models, achieving 63.8% accuracy. Our results suggest that the task is complex yet feasible, with many challenges ahead.

翻译：自动化事实核查（AFC）的证据数据可涵盖文本、表格、图像、音频或视频等多种模态。尽管利用图像进行AFC的研究日益增多，但现有工作主要聚焦于检测被篡改或伪造的图像。我们提出一项新任务——基于图表的事实核查，并引入ChartBERT作为首个针对图表证据的AFC模型。ChartBERT通过融合图表的文本、结构与视觉信息，判定文本主张的真实性。为进行评测，我们构建了包含15,886张图表的新数据集ChartFC。系统评估75种不同视觉语言（VL）基线模型后，结果表明ChartBERT以63.8%的准确率超越所有VL模型。研究显示该任务虽具可行性但复杂度高，未来仍面临诸多挑战。