Machine translation systems have been widely adopted in our daily life, making life easier and more convenient. Unfortunately, erroneous translations may result in severe consequences, such as financial losses. This requires to improve the accuracy and the reliability of machine translation systems. However, it is challenging to test machine translation systems because of the complexity and intractability of the underlying neural models. To tackle these challenges, we propose a novel metamorphic testing approach by syntactic tree pruning (STP) to validate machine translation systems. Our key insight is that a pruned sentence should have similar crucial semantics compared with the original sentence. Specifically, STP (1) proposes a core semantics-preserving pruning strategy by basic sentence structure and dependency relations on the level of syntactic tree representation; (2) generates source sentence pairs based on the metamorphic relation; (3) reports suspicious issues whose translations break the consistency property by a bag-of-words model. We further evaluate STP on two state-of-the-art machine translation systems (i.e., Google Translate and Bing Microsoft Translator) with 1,200 source sentences as inputs. The results show that STP can accurately find 5,073 unique erroneous translations in Google Translate and 5,100 unique erroneous translations in Bing Microsoft Translator (400% more than state-of-the-art techniques), with 64.5% and 65.4% precision, respectively. The reported erroneous translations vary in types and more than 90% of them cannot be found by state-of-the-art techniques. There are 9,393 erroneous translations unique to STP, which is 711.9% more than state-of-the-art techniques. Moreover, STP is quite effective to detect translation errors for the original sentences with a recall reaching 74.0%, improving state-of-the-art techniques by 55.1% on average.

翻译：机器翻译系统已广泛应用于日常生活，极大提升了便捷性。然而，错误的翻译可能导致严重后果（如经济损失），这要求提高机器翻译系统的准确性和可靠性。但受限于底层神经模型的复杂性和难解性，测试机器翻译系统仍面临挑战。为此，我们提出一种基于句法树剪枝（STP）的蜕变测试方法，用于验证机器翻译系统。核心思想在于：剪枝后的句子应与原始句子保持近似的关键语义。具体而言，STP (1) 基于句法树表征层级的基本句结构和依存关系，提出保留核心语义的剪枝策略；(2) 依据蜕变关系生成源句对；(3) 通过词袋模型标记破坏翻译一致性的可疑问题。我们在两个主流机器翻译系统（Google Translate和Bing Microsoft Translator）上使用1200个源句作为输入进行评估。结果表明，STP能准确发现Google Translate中的5073个独特错误翻译和Bing Microsoft Translator中的5100个独特错误翻译（较现有最优技术提升400%），精确率分别达到64.5%和65.4%。所报告的错误翻译类型多样，其中超过90%无法被现有最优技术检出。STP独有错误翻译数量达9393个，较现有最优技术提升711.9%。此外，STP对原始句子的翻译错误检测召回率达74.0%，平均较现有最优技术提升55.1%。