The paper studies the capabilities of Recurrent-Neural-Network sequence to sequence (RNN seq2seq) models in learning four transduction tasks: identity, reversal, total reduplication, and quadratic copying. These transductions are traditionally well studied under finite state transducers and attributed with increasing complexity. We find that RNN seq2seq models are only able to approximate a mapping that fits the training or in-distribution data, instead of learning the underlying functions. Although attention makes learning more efficient and robust, it does not overcome the out-of-distribution generalization limitation. We establish a novel complexity hierarchy for learning the four tasks for attention-less RNN seq2seq models, which may be understood in terms of the complexity hierarchy of formal languages, instead of string transductions. RNN variants also play a role in the results. In particular, we show that Simple RNN seq2seq models cannot count the input length.

翻译：本文研究了循环神经网络序列到序列（RNN seq2seq）模型在学习四种转换任务中的能力：恒等映射、反转、完全重复和二次复制。这些转换传统上在有限状态转换器框架下被充分研究，并被认为是复杂度递增的。我们发现RNN seq2seq模型仅能近似拟合训练数据或分布内数据的映射关系，而非真正学习底层函数。尽管注意力机制使学习更高效、更鲁棒，但仍无法克服分布外泛化的局限性。我们为无注意力机制的RNN seq2seq模型在学习这四项任务时建立了新的复杂度层级结构，该结构可从形式语言而非字符串转换的复杂度层级角度理解。此外，RNN变体对结果也有重要影响，特别是我们证明了简单RNN seq2seq模型无法计数输入长度。