Although the problem of hallucinations in neural machine translation (NMT) has received some attention, research on this highly pathological phenomenon lacks solid ground. Previous work has been limited in several ways: it often resorts to artificial settings where the problem is amplified, it disregards some (common) types of hallucinations, and it does not validate adequacy of detection heuristics. In this paper, we set foundations for the study of NMT hallucinations. First, we work in a natural setting, i.e., in-domain data without artificial noise neither in training nor in inference. Next, we annotate a dataset of over 3.4k sentences indicating different kinds of critical errors and hallucinations. Then, we turn to detection methods and both revisit methods used previously and propose using glass-box uncertainty-based detectors. Overall, we show that for preventive settings, (i) previously used methods are largely inadequate, (ii) sequence log-probability works best and performs on par with reference-based methods. Finally, we propose DeHallucinator, a simple method for alleviating hallucinations at test time that significantly reduces the hallucinatory rate. To ease future research, we release our annotated dataset for WMT18 German-English data, along with the model, training data, and code.

翻译：尽管神经机器翻译中的幻觉问题已引起一定关注，但针对这一高病理现象的研究仍缺乏坚实基础。现有工作存在多重局限：往往采用放大问题的人工设定场景，忽略某些（常见）幻觉类型，且未验证检测启发式方法的充分性。本文为神经机器翻译幻觉研究奠定基础。首先，我们基于自然场景开展研究，即训练和推理阶段均不含人工噪声的领域内数据。其次，我们标注了包含3400余个句子的数据集，标注出不同类型的严重错误与幻觉。继而，我们聚焦检测方法，不仅重新审视既有方法，还提出了基于透明箱不确定性指标的检测方案。总体而言，我们证明在预防性场景下：(i) 既有方法普遍存在缺陷；(ii) 序列对数概率方法表现最佳，其性能与基于参考的方法相当。最后，我们提出DeHallucinator——一种测试阶段缓解幻觉的简易方法，可显著降低幻觉率。为促进后续研究，我们开源了基于WMT18德英数据的标注数据集，并同步提供模型、训练数据及代码。