Despite the rising prevalence of neural sequence models, recent empirical evidences suggest their deficiency in compositional generalization. One of the current de-facto solutions to this problem is compositional data augmentation, aiming to incur additional compositional inductive bias. Nonetheless, the improvement offered by existing handcrafted augmentation strategies is limited when successful systematic generalization of neural sequence models requires multi-grained compositional bias (i.e., not limited to either lexical or structural biases only) or differentiation of training sequences in an imbalanced difficulty distribution. To address the two challenges, we first propose a novel compositional augmentation strategy dubbed \textbf{Span} \textbf{Sub}stitution (SpanSub) that enables multi-grained composition of substantial substructures in the whole training set. Over and above that, we introduce the \textbf{L}earning \textbf{to} \textbf{S}ubstitute \textbf{S}pan (L2S2) framework which empowers the learning of span substitution probabilities in SpanSub in an end-to-end manner by maximizing the loss of neural sequence models, so as to outweigh those challenging compositions with elusive concepts and novel surroundings. Our empirical results on three standard compositional generalization benchmarks, including SCAN, COGS and GeoQuery (with an improvement of at most 66.5\%, 10.3\%, 1.2\%, respectively), demonstrate the superiority of SpanSub, %the learning framework L2S2 and their combination.

翻译：尽管神经序列模型日益普及，但近期经验证据表明其在组合泛化方面存在不足。针对该问题的当前主流解决方案之一是组合数据增强，旨在引入额外的组合归纳偏置。然而，当神经序列模型成功实现系统性泛化需要多粒度组合偏置（即不仅限于词汇或结构偏置）或需区分难度分布不均衡的训练序列时，现有手工设计的增强策略带来的改进效果仍有限。为应对这两项挑战，我们首先提出一种名为**跨度替换**（SpanSub）的新型组合增强策略，该策略能够对整个训练集中大量子结构进行多粒度组合。在此基础上，我们引入**学习替换跨度**（L2S2）框架，该框架通过最大化神经序列模型的损失，以端到端方式学习SpanSub中的跨度替换概率，从而为那些包含隐晦概念及新颖组合环境的困难样本赋予更高权重。我们在三个标准组合泛化基准（包括SCAN、COGS和GeoQuery，分别提升最高达66.5%、10.3%、1.2%）上的实验结果表明了SpanSub、学习框架L2S2及其组合的优越性。