Transformers use the dense self-attention mechanism which gives a lot of flexibility for long-range connectivity. Over multiple layers of a deep transformer, the number of possible connectivity patterns increases exponentially. However, very few of these contribute to the performance of the network, and even fewer are essential. We hypothesize that there are sparsely connected sub-networks within a transformer, called information pathways which can be trained independently. However, the dynamic (i.e., input-dependent) nature of these pathways makes it difficult to prune dense self-attention during training. But the overall distribution of these pathways is often predictable. We take advantage of this fact to propose Stochastically Subsampled self-Attention (SSA) - a general-purpose training strategy for transformers that can reduce both the memory and computational cost of self-attention by 4 to 8 times during training while also serving as a regularization method - improving generalization over dense training. We show that an ensemble of sub-models can be formed from the subsampled pathways within a network, which can achieve better performance than its densely attended counterpart. We perform experiments on a variety of NLP, computer vision and graph learning tasks in both generative and discriminative settings to provide empirical evidence for our claims and show the effectiveness of the proposed method.

翻译：Transformer采用密集自注意力机制，为长程连接提供了高度灵活性。在深层Transformer的多个层中，可能的连接模式数量呈指数级增长。然而，其中极少部分对网络性能有贡献，甚至更少部分是关键的。我们提出假设：Transformer内部存在稀疏连接的子网络，称为信息路径，这些路径可以独立训练。然而，这些路径的动态（即依赖于输入）特性使得在训练期间难以剪枝密集自注意力机制。但这些路径的整体分布通常是可预测的。我们利用这一事实提出随机子采样自注意力（SSA）——一种通用的Transformer训练策略，可将训练期间自注意力的内存和计算成本降低4至8倍，同时作为正则化方法提升泛化性能，优于密集训练。我们证明，从网络内子采样路径中可以形成一个子模型集成，其性能优于对应的密集注意力模型。我们在自然语言处理、计算机视觉和图学习任务的生成式与判别式场景中进行了实验，为上述观点提供实证依据，并展示所提方法的有效性。