Fact-checking is essential due to the explosion of misinformation in the media ecosystem. Although false information exists in every language and country, most research to solve the problem mainly concentrated on huge communities like English and Chinese. Low-resource languages like Vietnamese are necessary to explore corpora and models for fact verification. To bridge this gap, we construct ViWikiFC, the first manual annotated open-domain corpus for Vietnamese Wikipedia Fact Checking more than 20K claims generated by converting evidence sentences extracted from Wikipedia articles. We analyze our corpus through many linguistic aspects, from the new dependency rate, the new n-gram rate, and the new word rate. We conducted various experiments for Vietnamese fact-checking, including evidence retrieval and verdict prediction. BM25 and InfoXLM (Large) achieved the best results in two tasks, with BM25 achieving an accuracy of 88.30% for SUPPORTS, 86.93% for REFUTES, and only 56.67% for the NEI label in the evidence retrieval task, InfoXLM (Large) achieved an F1 score of 86.51%. Furthermore, we also conducted a pipeline approach, which only achieved a strict accuracy of 67.00% when using InfoXLM (Large) and BM25. These results demonstrate that our dataset is challenging for the Vietnamese language model in fact-checking tasks.
翻译:事实核查对于应对媒体生态系统中虚假信息的爆炸性增长至关重要。尽管虚假信息存在于各种语言和国家,但相关研究主要集中于英语和汉语等大型语言社群。对于越南语等低资源语言,亟需开发用于事实核查的语料库和模型。为填补这一空白,我们构建了ViWikiFC——首个针对越南语维基百科的手动标注开放领域事实核查语料库,包含超过2万条通过转换维基百科证据句生成的声明。我们从新依存关系率、新n-gram率及新词率等多个语言学维度对语料库进行了分析。针对越南语事实核查任务,我们开展了包含证据检索与判定预测的多组实验。BM25与InfoXLM(Large)在两项任务中取得最佳效果:在证据检索任务中,BM25对SUPPORTS标签的准确率达88.30%,对REFUTES标签达86.93%,而对NEI标签仅为56.67%;InfoXLM(Large)在判定预测任务中F1分数达86.51%。此外,我们实施的流水线方法在使用InfoXLM(Large)与BM25组合时,严格准确率仅达到67.00%。这些结果表明,我们的数据集对越南语模型在事实核查任务中构成了显著挑战。