Modelling a language model for a multi-lingual scenario includes several potential challenges, among which catastrophic forgetting is the major challenge. For example, small language models (SLM) built for low-resource languages by adapting large language models (LLMs) pose the challenge of catastrophic forgetting. This work proposes to employ a continual learning strategy using parts-of-speech (POS)-based code-switching along with a replay adapter strategy to mitigate the identified gap of catastrophic forgetting while training SLM from LLM. Experiments conducted on vision language tasks such as visual question answering and language modelling task exhibits the success of the proposed architecture.

翻译：在多语言场景下构建语言模型面临若干潜在挑战，其中灾难性遗忘是主要难题。例如，通过适配大语言模型（LLM）为低资源语言构建的小语言模型（SLM）便存在灾难性遗忘的挑战。本研究提出采用基于词性（POS）的语码转换与回放适配器相结合的持续学习策略，以缓解在利用LLM训练SLM时存在的灾难性遗忘问题。在视觉问答等视觉语言任务及语言建模任务上的实验表明，所提出的架构取得了成功。