We explore different ways to utilize position-based cross-attention in seq2seq networks to enable length generalization in algorithmic tasks. We show that a simple approach of interpolating the original and reversed encoded representations combined with relative attention allows near-perfect length generalization for both forward and reverse lookup tasks or copy tasks that had been generally hard to tackle. We also devise harder diagnostic tasks where the relative distance of the ideal attention position varies with timestep. In such settings, the simple interpolation trick with relative attention is not sufficient. We introduce novel variants of location attention building on top of Dubois et al. (2020) to address the new diagnostic tasks. We also show the benefits of our approaches for length generalization in SCAN (Lake & Baroni, 2018) and CFQ (Keysers et al., 2020). Our code is available on GitHub.

翻译：我们探索了在序列到序列网络中利用基于位置的交叉注意力实现算法任务长度泛化的不同方法。研究表明，通过对原始编码表示和逆序编码表示进行插值并结合相对注意力这一简单方法，能够在通常难以处理的正向查找、逆向查找及复制任务中近乎完美地实现长度泛化。我们进一步设计了更困难的诊断性任务：其中理想注意力位置的相对距离随时间步动态变化。在此类设置下，简单的插值技巧与相对注意力已不足以解决问题。我们在Dubois等人(2020)研究基础上提出了新型位置注意力变体以应对这些新诊断任务。同时验证了所提方法在SCAN (Lake & Baroni, 2018)和CFQ (Keysers et al., 2020)数据集上对长度泛化的促进作用。我们的代码已开源至GitHub。