Adapters have emerged as a modular and parameter-efficient approach to (zero-shot) cross-lingual transfer. The established MAD-X framework employs separate language and task adapters which can be arbitrarily combined to perform the transfer of any task to any target language. Subsequently, BAD-X, an extension of the MAD-X framework, achieves improved transfer at the cost of MAD-X's modularity by creating "bilingual" adapters specific to the source-target language pair. In this work, we aim to take the best of both worlds by (i) fine-tuning task adapters adapted to the target language(s) (so-called "target language-ready" (TLR) adapters) to maintain high transfer performance, but (ii) without sacrificing the highly modular design of MAD-X. The main idea of "target language-ready" adapters is to resolve the training-vs-inference discrepancy of MAD-X: the task adapter "sees" the target language adapter for the very first time during inference, and thus might not be fully compatible with it. We address this mismatch by exposing the task adapter to the target language adapter during training, and empirically validate several variants of the idea: in the simplest form, we alternate between using the source and target language adapters during task adapter training, which can be generalized to cycling over any set of language adapters. We evaluate different TLR-based transfer configurations with varying degrees of generality across a suite of standard cross-lingual benchmarks, and find that the most general (and thus most modular) configuration consistently outperforms MAD-X and BAD-X on most tasks and languages.

翻译：适配器已作为一种模块化且参数高效的方法出现，用于（零样本）跨语言迁移。成熟的MAD-X框架采用独立的语言适配器和任务适配器，可任意组合以将任何任务迁移至任何目标语言。随后，作为MAD-X框架的扩展，BAD-X通过创建特定于源-目标语言对的“双语”适配器，以牺牲MAD-X的模块化为代价实现了改进的迁移性能。在本工作中，我们旨在取两者之长：（i）微调适配于目标语言的任务适配器（即所谓的“目标语言就绪”（TLR）适配器），以保持高迁移性能，同时（ii）不牺牲MAD-X的高度模块化设计。“目标语言就绪”适配器的主要思想是解决MAD-X训练与推理之间的不一致性：任务适配器在推理过程中首次“看到”目标语言适配器，因此可能与其不完全兼容。我们通过让任务适配器在训练期间暴露于目标语言适配器来解决这一不匹配问题，并实证验证了该思想的若干变体：在最简形式中，我们在任务适配器训练期间交替使用源语言适配器和目标语言适配器，这可以推广到对任意语言适配器集合进行循环训练。我们评估了不同TLR迁移配置（具有不同程度的通用性）在一系列标准跨语言基准上的表现，发现最通用（因此也最具模块化）的配置在大多数任务和语言上持续优于MAD-X和BAD-X。