Hallucinations pose a significant challenge to the reliability of neural models for abstractive summarisation. While automatically generated summaries may be fluent, they often lack faithfulness to the original document. This issue becomes even more pronounced in low-resource settings, such as cross-lingual transfer. With the existing faithful metrics focusing on English, even measuring the extent of this phenomenon in cross-lingual settings is hard. To address this, we first develop a novel metric, mFACT, evaluating the faithfulness of non-English summaries, leveraging translation-based transfer from multiple English faithfulness metrics. We then propose a simple but effective method to reduce hallucinations with a cross-lingual transfer, which weighs the loss of each training example by its faithfulness score. Through extensive experiments in multiple languages, we demonstrate that mFACT is the metric that is most suited to detect hallucinations. Moreover, we find that our proposed loss weighting method drastically increases both performance and faithfulness according to both automatic and human evaluation when compared to strong baselines for cross-lingual transfer such as MAD-X. Our code and dataset are available at https://github.com/yfqiu-nlp/mfact-summ.

翻译：幻觉现象对神经模型在抽象式摘要中的可靠性构成了重大挑战。虽然自动生成的摘要可能流畅，但往往缺乏对原始文档的忠实度。这一问题在低资源场景下（如跨语言迁移）更为突出。由于现有忠实度指标主要针对英文，即使在跨语言场景中衡量这一现象的程度也较为困难。为解决这一问题，我们首先提出一种新指标mFACT，通过基于翻译的多种英文忠实度指标迁移，评估非英语摘要的忠实度。随后我们提出一种简单但有效的跨语言迁移方法，通过根据训练样本的忠实度得分对其损失进行加权来减少幻觉。通过多语言的广泛实验，我们证明mFACT是最适合检测幻觉的指标。此外，我们发现与MAD-X等强跨语言迁移基线相比，我们提出的损失加权方法在自动评估和人工评估中均显著提升了模型性能与忠实度。我们的代码和数据集已开源至https://github.com/yfqiu-nlp/mfact-summ。