Recent work in incremental learning has introduced diverse approaches to tackle catastrophic forgetting from data augmentation to optimized training regimes. However, most of them focus on very few training steps. We propose a method for robust incremental learning over dozens of fine-tuning steps using data from a variety of languages. We show that a combination of data-augmentation and an optimized training regime allows us to continue improving the model even for as many as fifty training steps. Crucially, our augmentation strategy does not require retaining access to previous training data and is suitable in scenarios with privacy constraints.

翻译：近来的渐进学习工作引入了多种方法,以解决从数据扩充到优化培训制度等灾难性的遗忘问题,但多数侧重于极少数培训步骤。我们提出了一个方法,利用各种语言的数据,对数十个微调步骤进行有力的渐进学习。我们表明,数据扩充和优化培训制度相结合,使我们能够继续改进模式,即使有多达50个培训步骤。关键是,我们的增强战略并不要求保留先前的培训数据,而且适合隐私限制的情况。