Recent studies have shown that sequence-to-sequence (seq2seq) models struggle with compositional generalization (CG), i.e., the ability to systematically generalize to unseen compositions of seen components. There is mounting evidence that one of the reasons hindering CG is the representation of the encoder uppermost layer is entangled, i.e., the syntactic and semantic representations of sequences are entangled. However, we consider that the previously identified representation entanglement problem is not comprehensive enough. Additionally, we hypothesize that the source keys and values representations passing into different decoder layers are also entangled. Starting from this intuition, we propose \textsc{CompoSition} (\textbf{Compo}se \textbf{S}yntactic and Semant\textbf{i}c Representa\textbf{tion}s), an extension to seq2seq models which learns to compose representations of different encoder layers dynamically for different tasks, since recent studies reveal that the bottom layers of the Transformer encoder contain more syntactic information and the top ones contain more semantic information. Specifically, we introduce a \textit{composed layer} between the encoder and decoder to compose different encoder layers' representations to generate specific keys and values passing into different decoder layers. \textsc{CompoSition} achieves competitive results on two comprehensive and realistic benchmarks, which empirically demonstrates the effectiveness of our proposal. Codes are available at~\url{https://github.com/thinkaboutzero/COMPOSITION}.

翻译：近期研究表明，序列到序列（seq2seq）模型在组合泛化（CG）方面存在困难，即系统性地泛化至未见过的已见组件组合的能力。越来越多的证据表明，阻碍组合泛化的原因之一是编码器顶层表示的纠缠性，即序列的句法和语义表示相互纠缠。然而，我们认为先前识别的表示纠缠问题尚不够全面。此外，我们假设传入不同解码器层的源键和值表示也存在纠缠。基于这一直觉，我们提出\textsc{CompoSition}（\textbf{组合}句法和语义\textbf{表示}），这是对seq2seq模型的一种扩展，它学习针对不同任务动态组合不同编码器层的表示，因为近期研究表明Transformer编码器的底层包含更多句法信息，而顶层包含更多语义信息。具体而言，我们在编码器和解码器之间引入一个\textit{组合层}，用于组合不同编码器层的表示以生成传入不同解码器层的特定键和值。\textsc{CompoSition}在两个全面且真实的基准测试中取得了具有竞争力的结果，这实证性地证明了我们方法的有效性。代码可在~\url{https://github.com/thinkaboutzero/COMPOSITION} 获取。