Prior work shows that it is possible to expand pretrained Masked Language Models (MLMs) to new languages by learning a new set of embeddings, while keeping the transformer body frozen. Despite learning a small subset of parameters, this approach is not compute-efficient, as training the new embeddings requires a full forward and backward pass over the entire model. We propose mini-model adaptation, a compute-efficient alternative that builds a shallow mini-model from a fraction of a large model's parameters. New language-specific embeddings can then be efficiently trained over the mini-model and plugged into the aligned large model for rapid cross-lingual transfer. We explore two approaches to learn mini-models: MiniJoint, which jointly pretrains the primary model and the mini-model using a single transformer with a secondary MLM head at a middle layer; and MiniPost, where we start from a regular pretrained model, build a mini-model by extracting and freezing a few layers, and learn a small number of parameters on top. Experiments on XNLI, MLQA and PAWS-X show that mini-model adaptation matches the performance of the standard approach using 2.3x less compute on average.

翻译：先前工作表明，通过学习一组新嵌入并冻结Transformer主体，可以将预训练掩码语言模型（MLM）扩展至新语言。尽管仅需学习少量参数，该方法计算效率不高，因为训练新嵌入需要对整个模型进行完整的前向和反向传播。我们提出小模型适配（mini-model adaptation），这是一种计算效率更高的替代方案，从大模型参数的一小部分中构建浅层小模型。随后，新语言特定嵌入可基于小模型高效训练，并插入对齐的大模型中，实现快速跨语言迁移。我们探索了两种小模型学习方法：MiniJoint——使用单个Transformer在中间层增设辅助MLM头，联合预训练主模型与小模型；以及MiniPost——从常规预训练模型出发，提取并冻结若干层构建小模型，并在其顶层学习少量参数。在XNLI、MLQA和PAWS-X上的实验表明，小模型适配在平均降低2.3倍计算量的同时，达到了与标准方法相当的性能。