The ability to learn new concepts sequentially is a major weakness for modern neural networks, which hinders their use in non-stationary environments. Their propensity to fit the current data distribution to the detriment of the past acquired knowledge leads to the catastrophic forgetting issue. In this work we tackle the problem of Spoken Language Understanding applied to a continual learning setting. We first define a class-incremental scenario for the SLURP dataset. Then, we propose three knowledge distillation (KD) approaches to mitigate forgetting for a sequence-to-sequence transformer model: the first KD method is applied to the encoder output (audio-KD), and the other two work on the decoder output, either directly on the token-level (tok-KD) or on the sequence-level (seq-KD) distributions. We show that the seq-KD substantially improves all the performance metrics, and its combination with the audio-KD further decreases the average WER and enhances the entity prediction metric.

翻译：现代神经网络在顺序学习新概念方面存在重大缺陷，这阻碍了其在非平稳环境中的应用。其倾向于适配当前数据分布而损害先前获取知识的特性，导致了灾难性遗忘问题。本研究针对持续学习场景下的口语理解任务，首先为SLURP数据集定义了类别增量场景，进而提出三种知识蒸馏方法以缓解序列到序列Transformer模型的遗忘：第一种KD方法作用于编码器输出（音频-KD），另外两种作用于解码器输出，分别针对令牌级（tok-KD）和序列级（seq-KD）分布。实验表明，序列级KD显著提升了所有性能指标，且与音频-KD的结合进一步降低了平均词错误率并提升了实体预测指标。