Creating neural text encoders for written Swiss German is challenging due to a dearth of training data combined with dialectal variation. In this paper, we build on several existing multilingual encoders and adapt them to Swiss German using continued pre-training. Evaluation on three diverse downstream tasks shows that simply adding a Swiss German adapter to a modular encoder achieves 97.5% of fully monolithic adaptation performance. We further find that for the task of retrieving Swiss German sentences given Standard German queries, adapting a character-level model is more effective than the other adaptation strategies. We release our code and the models trained for our experiments at https://github.com/ZurichNLP/swiss-german-text-encoders

翻译：为瑞士书面德语创建神经文本编码器面临挑战，主要源于训练数据匮乏及方言变体多样性。本文基于现有多种多语言编码器，通过持续预训练方法将其适配至瑞士德语。在三个不同下游任务上的评估表明，仅需向模块化编码器添加瑞士德语适配器，即可达到全单块适配性能的97.5%。进一步研究发现，在根据标准德语查询检索瑞士德语语句的任务中，字符级模型的适配效果优于其他适配策略。我们在https://github.com/ZurichNLP/swiss-german-text-encoders 开源了实验代码与训练模型。