Structured dropout approaches, such as attention dropout and DropHead, have been investigated to regularize the multi-head attention mechanism in Transformers. In this paper, we propose a new regularization scheme based on token-level rather than structure-level to reduce overfitting. Specifically, we devise a novel Token-Level Masking (TLM) training strategy for Transformers to regularize the connections of self-attention, which consists of two masking techniques that are effective and easy to implement. The underlying idea is to manipulate the connections between tokens in the multi-head attention via masking, where the networks are forced to exploit partial neighbors' information to produce a meaningful representation. The generality and effectiveness of TLM are thoroughly evaluated via extensive experiments on 4 diversified NLP tasks across 18 datasets, including natural language understanding benchmark GLUE, ChineseGLUE, Chinese Grammatical Error Correction, and data-to-text generation. The results indicate that TLM can consistently outperform attention dropout and DropHead, e.g., it increases by 0.5 points relative to DropHead with BERT-large on GLUE. Moreover, TLM can establish a new record on the data-to-text benchmark Rotowire (18.93 BLEU). Our code will be publicly available at https://github.com/Young1993/tlm.

翻译：结构化dropout方法（如注意力dropout和DropHead）已被研究用于正则化Transformer中的多头注意力机制。本文提出一种基于token级别而非结构级别的新型正则化方案以降低过拟合。具体而言，我们为Transformer设计了一种新颖的Token级掩码（TLM）训练策略，通过两种易于实现且有效的掩码技术正则化自注意力连接。其核心思想是通过掩码操作调控多头注意力中token之间的连接，迫使网络利用局部邻居信息生成有意义的表征。通过在涵盖自然语言理解基准GLUE、ChineseGLUE、中文语法纠错及数据到文本生成等4类自然语言处理任务的18个数据集上的广泛实验，全面验证了TLM的通用性与有效性。结果表明，TLM始终优于注意力dropout和DropHead——例如，在GLUE上基于BERT-large模型，TLM相比DropHead提升0.5个百分点。此外，TLM在数据到文本基准Rotowire上创下新纪录（18.93 BLEU值）。我们的代码将在https://github.com/Young1993/tlm 开源。