Adapter modules, additional trainable parameters that enable efficient fine-tuning of pretrained transformers, have recently been used for language specialization of multilingual transformers, improving downstream zero-shot cross-lingual transfer. In this work, we propose orthogonal language and task adapters (dubbed orthoadapters) for cross-lingual transfer. They are trained to encode language- and task-specific information that is complementary (i.e., orthogonal) to the knowledge already stored in the pretrained transformer's parameters. Our zero-shot cross-lingual transfer experiments, involving three tasks (POS-tagging, NER, NLI) and a set of 10 diverse languages, 1) point to the usefulness of orthoadapters in cross-lingual transfer, especially for the most complex NLI task, but also 2) indicate that the optimal adapter configuration highly depends on the task and the target language. We hope that our work will motivate a wider investigation of usefulness of orthogonality constraints in language- and task-specific fine-tuning of pretrained transformers.

翻译：在这项工作中,我们提议使用正方位语言和任务调适器(dubbed orthodapterers)进行跨语种转让,并提议采用正方位语言和任务调适器(dubbbed orthodapters)进行跨语种转让,这些调适器是经过培训的变压器能够有效微调变压器的额外可培训参数,最近用于多语种变压器的语言专业化,改进下游零点跨语言调试,改进下游零点跨语言的跨语言调试。我们建议采用正方位语言语言语言和任务调适器进行跨语言调用,特别是对于最复杂的国家变压器任务和目标语言而言,但2 表明最佳调适配器配置在很大程度上取决于任务和目标语言。我们希望我们的工作能够激发对预先培训变压器在语言和具体任务调整方面受到的异度限制的实用性进行更广泛的调查。