Multi-head attention is appealing for the ability to jointly attend to information from different representation subspaces at different positions. In this work, we introduce a disagreement regularization to explicitly encourage the diversity among multiple attention heads. Specifically, we propose three types of disagreement regularization, which respectively encourage the subspace, the attended positions, and the output representation associated with each attention head to be different from other heads. Experimental results on widely-used WMT14 English-German and WMT17 Chinese-English translation tasks demonstrate the effectiveness and universality of the proposed approach.

翻译：多股关注要求能够共同处理不同代表分空间在不同职位上的信息,在这项工作中,我们引入了不同意见规范化,明确鼓励多重关注负责人之间的多样性。具体地说,我们提出了三种分歧规范化类型,分别鼓励分空间、参与职位和与每个关注负责人相关的产出代表与其它负责人不同。 广泛使用的WMT14英文-德文翻译和WMT17中文-英文翻译任务的实验结果显示了拟议方法的有效性和普遍性。