We present SwissBERT, a masked language model created specifically for processing Switzerland-related text. SwissBERT is a pre-trained model that we adapted to news articles written in the national languages of Switzerland -- German, French, Italian, and Romansh. We evaluate SwissBERT on natural language understanding tasks related to Switzerland and find that it tends to outperform previous models on these tasks, especially when processing contemporary news and/or Romansh Grischun. Since SwissBERT uses language adapters, it may be extended to Swiss German dialects in future work. The model and our open-source code are publicly released at https://github.com/ZurichNLP/swissbert.

翻译：我们提出了SwissBERT，一个专为处理与瑞士相关的文本而设计的掩码语言模型。SwissBERT是一个预训练模型，我们针对瑞士官方语言——德语、法语、意大利语及罗曼什语的新闻文章进行了适配。我们在与瑞士相关的自然语言理解任务上对SwissBERT进行了评估，发现它在这些任务上往往优于先前的模型，尤其是在处理当代新闻和/或罗曼什语Grischun时。由于SwissBERT使用了语言适配器，未来可将其扩展至瑞士德语方言。该模型及我们的开源代码已公开发布于https://github.com/ZurichNLP/swissbert。