Despite the rapid recent progress in creating accurate and compact in-context learners, most recent work focuses on in-context learning (ICL) for tasks in English. However, the ability to interact with users of languages outside English presents a great potential for broadening the applicability of language technologies to non-English speakers. In this work, we collect the infrastructure necessary for training and evaluation of ICL in a selection of Slavic languages: Czech, Polish, and Russian. We link a diverse set of datasets and cast these into a unified instructional format through a set of transformations and newly-crafted templates written purely in target languages. Using the newly-curated dataset, we evaluate a set of the most recent in-context learners and compare their results to the supervised baselines. Finally, we train, evaluate and publish a set of in-context learning models that we train on the collected resources and compare their performance to previous work. We find that ICL models tuned in English are also able to learn some tasks from non-English contexts, but multilingual instruction fine-tuning consistently improves the ICL ability. We also find that the massive multitask training can be outperformed by single-task training in the target language, uncovering the potential for specializing in-context learners to the language(s) of their application.

翻译：尽管近期在构建精准且紧凑的情境学习者方面取得了快速进展，但大多数最新研究仍聚焦于英语任务的情境学习（ICL）。然而，与非英语使用者进行语言交互的能力，为拓展语言技术在非英语人群中的应用提供了巨大潜力。本研究收集了训练和评估斯拉夫语言（捷克语、波兰语和俄语）情境学习所需的基础设施。我们整合了多样化的数据集，并通过一系列变换和新设计的纯目标语言模板，将其统一转化为指令式格式。利用新整理的数据集，我们评估了多款最新的情境学习者，并将其结果与有监督基线进行了比较。最终，我们训练、评估并发布了一系列基于所收集资源训练的情境学习模型，并将其性能与先前工作进行了对比。研究发现，英语调优的情境学习模型也能从非英语语境中学习部分任务，但多语言指令微调能持续提升情境学习能力。同时，我们还发现，在目标语言上进行单任务训练可超越大规模多任务训练，这揭示了将情境学习者专门应用于特定语言（或多种语言）的巨大潜力。