Neural Machine Translation (NMT) models are state-of-the-art for machine translation. However, these models are known to have various social biases, especially gender bias. Most of the work on evaluating gender bias in NMT has focused primarily on English as the source language. For source languages different from English, most of the studies use gender-neutral sentences to evaluate gender bias. However, practically, many sentences that we encounter do have gender information. Therefore, it makes more sense to evaluate for bias using such sentences. This allows us to determine if NMT models can identify the correct gender based on the grammatical gender cues in the source sentence rather than relying on biased correlations with, say, occupation terms. To demonstrate our point, in this work, we use Hindi as the source language and construct two sets of gender-specific sentences: OTSC-Hindi and WinoMT-Hindi that we use to evaluate different Hindi-English (HI-EN) NMT systems automatically for gender bias. Our work highlights the importance of considering the nature of language when designing such extrinsic bias evaluation datasets.

翻译：神经机器翻译（NMT）模型是机器翻译领域的尖端技术。然而，这些模型已知存在各种社会偏见，尤其是性别偏见。目前评估NMT中性别偏见的绝大部分工作主要聚焦于英语作为源语言的情况。对于非英语源语言，大多数研究采用性别中立句子来评估性别偏见。但在实际应用中，我们遇到的大量句子确实包含性别信息。因此，利用此类句子进行偏见评估更具实际意义。这使我们能够判断NMT模型是否能够基于源句中的语法性别线索识别正确性别，而非依赖与职业术语等概念的偏见性关联。为验证这一观点，本研究以印地语为源语言，构建了两组特定性别句子集：OTSC-印地语和WinoMT-印地语，并据此对多个印地语-英语（HI-EN）NMT系统进行自动化性别偏见评估。本研究凸显了在设计此类外在偏见评估数据集时需充分考虑语言特性的重要性。