Machine translation often suffers from biased data and algorithms that can lead to unacceptable errors in system output. While bias in gender norms has been investigated, less is known about whether MT systems encode bias about social relationships, e.g. sentences such as "the lawyer kissed her wife." We investigate the degree of bias against same-gender relationships in MT systems, using generated template sentences drawn from several noun-gender languages (e.g. Spanish). We find that three popular MT services consistently fail to accurately translate sentences concerning relationships between nouns of the same gender. The error rate varies considerably based on the context, e.g. same-gender sentences referencing high female-representation occupations are translated with lower accuracy. We provide this work as a case study in the evaluation of intrinsic bias in NLP systems, with respect to social relationships.

翻译：机器翻译常因数据与算法中的偏见导致系统输出出现不可接受的错误。尽管性别规范偏见已有研究，但对机器翻译系统是否编码社会关系偏见（例如"律师亲吻了她的妻子"这类句子）仍知之甚少。我们利用从多种名词性语言（如西班牙语）生成的模板句子，考察了机器翻译系统中针对同性关系的偏见程度。研究发现，三种主流机器翻译服务在翻译涉及同性别名词间关系的句子时，始终无法准确输出。错误率因语境差异显著，例如提及女性高比例职业的同性关系句子翻译准确率更低。本文将此作为自然语言处理系统内在偏见评估的案例研究，聚焦社会关系维度。