Existing neural models are demonstrated to struggle with compositional generalization (CG), i.e., the ability to systematically generalize to unseen compositions of seen components. A key reason for failure on CG is that the syntactic and semantic representations of sequences in both the uppermost layer of the encoder and decoder are entangled. However, previous work concentrates on separating the learning of syntax and semantics instead of exploring the reasons behind the representation entanglement (RE) problem to solve it. We explain why it exists by analyzing the representation evolving mechanism from the bottom to the top of the Transformer layers. We find that the ``shallow'' residual connections within each layer fail to fuse previous layers' information effectively, leading to information forgetting between layers and further the RE problems. Inspired by this, we propose LRF, a novel \textbf{L}ayer-wise \textbf{R}epresentation \textbf{F}usion framework for CG, which learns to fuse previous layers' information back into the encoding and decoding process effectively through introducing a \emph{fuse-attention module} at each encoder and decoder layer. LRF achieves promising results on two realistic benchmarks, empirically demonstrating the effectiveness of our proposal.

翻译：现有神经模型被证明难以实现组合泛化（CG），即对已知组件未见组合进行系统性泛化的能力。导致CG失败的关键原因在于编码器和解码器最顶层序列的句法与语义表示存在纠缠。然而，以往研究集中于分离句法与语义的学习，而非探索表示纠缠（RE）问题的根源以解决该问题。我们通过分析Transformer层中从底层到顶层的表示演化机制，解释了RE问题存在的原因。研究发现，各层内“浅层”残差连接未能有效融合前层信息，导致层间信息遗忘并进一步引发RE问题。受此启发，我们提出LRF——一种面向CG的新型逐层表示融合框架。该框架通过在每层编码器和解码器中引入融合注意力模块，学习将前层信息有效融合回编码与解码过程。LRF在两个真实基准测试中取得了优异结果，从实证角度验证了我们方案的有效性。