Prior work has attempted to understand the internal structures and functionalities of Transformer-based encoder-decoder architectures on the level of multi-head attention and feed-forward sublayers. Interpretations have focused on the encoder and decoder, along with the combinatorial possibilities of the self-attention, cross-attention, and feed-forward sublayers. However, without examining the low-level structures, one gains limited understanding of the motivation behind sublayer reordering. Could we dive into the sublayer abstraction and permute layer weight matrices to improve the quality of translation? We propose AEIUOrder to greedily reorder layer weight matrices in the encoder by their well-trainedness, as measured by Heavy-Tailed Self-Regularization (HT-SR) metrics, and order the decoder matrices correspondingly. Our results suggest that greedily reordering layer weight matrices to maximize Total well-trainedness facilitates the model to learn representations and generate translations more effectively.

翻译：先前的工作试图从多头注意力和前馈子层的层面理解基于变换器的编码器-解码器架构的内部结构与功能。这些解释聚焦于编码器与解码器，以及自注意力、交叉注意力和前馈子层的组合可能性。然而，若不对底层结构进行审视，便难以深入理解子层重排背后的动机。我们能否深入子层抽象层面，通过置换层权重矩阵来提升翻译质量？我们提出AEIUOrder方法，根据重尾自正则化（HT-SR）指标所衡量的良好训练程度，贪婪地对编码器中的层权重矩阵进行重排序，并相应地对解码器矩阵进行排序。我们的结果表明，通过贪婪地重排层权重矩阵以最大化总体良好训练程度，有助于模型更有效地学习表示并生成翻译。