Pre-trained Language Models (LMs) have become an integral part of Natural Language Processing (NLP) in recent years, due to their superior performance in downstream applications. In spite of this resounding success, the usability of LMs is constrained by computational and time complexity, along with their increasing size; an issue that has been referred to as `overparameterisation'. Different strategies have been proposed in the literature to alleviate these problems, with the aim to create effective compact models that nearly match the performance of their bloated counterparts with negligible performance losses. One of the most popular techniques in this area of research is model distillation. Another potent but underutilised technique is cross-layer parameter sharing. In this work, we combine these two strategies and present MiniALBERT, a technique for converting the knowledge of fully parameterised LMs (such as BERT) into a compact recursive student. In addition, we investigate the application of bottleneck adapters for layer-wise adaptation of our recursive student, and also explore the efficacy of adapter tuning for fine-tuning of compact models. We test our proposed models on a number of general and biomedical NLP tasks to demonstrate their viability and compare them with the state-of-the-art and other existing compact models. All the codes used in the experiments are available at https://github.com/nlpie-research/MiniALBERT. Our pre-trained compact models can be accessed from https://huggingface.co/nlpie.

翻译：近年来，预训练语言模型因其在下游应用中的卓越性能，已成为自然语言处理领域不可或缺的组成部分。尽管取得了巨大成功，但语言模型的可使用性受限于计算和时间复杂度，以及其日益增长的规模——这一问题被称为"过参数化"。为缓解这些挑战，文献中提出了多种策略，旨在创建性能接近臃肿模型且性能损失可忽略的有效紧凑模型。该研究领域最流行的技术之一是模型蒸馏，另一种强大但未被充分利用的技术是跨层参数共享。在本工作中，我们结合了这两种策略，提出了MiniALBERT——一种将全参数化语言模型（如BERT）的知识转化为紧凑递归学生模型的技术。此外，我们研究了在递归学生模型中应用瓶颈适配器进行逐层适配，并探讨了适配器调优对紧凑模型微调的有效性。我们在多项通用和生物医学自然语言处理任务上测试了所提模型，以验证其可行性，并与当前最先进及其他现有紧凑模型进行了比较。实验中使用的所有代码可在https://github.com/nlpie-research/MiniALBERT获取。我们预训练的紧凑模型可从https://huggingface.co/nlpie访问。