In this paper, we introduce the range of oBERTa language models, an easy-to-use set of language models, which allows Natural Language Processing (NLP) practitioners to obtain between 3.8 and 24.3 times faster models without expertise in model compression. Specifically, oBERTa extends existing work on pruning, knowledge distillation, and quantization and leverages frozen embeddings to improve knowledge distillation, and improved model initialization to deliver higher accuracy on a a broad range of transfer tasks. In generating oBERTa, we explore how the highly optimized RoBERTa differs from the BERT with respect to pruning during pre-training and fine-tuning and find it less amenable to compression during fine-tuning. We explore the use of oBERTa on a broad seven representative NLP tasks and find that the improved compression techniques allow a pruned oBERTa model to match the performance of BERTBASE and exceed the performance of Prune OFA Large on the SQUAD V1.1 Question Answering dataset, despite being 8x and 2x, respectively, faster in inference. We release our code, training regimes, and associated model for broad usage to encourage usage and experimentation.

翻译：本文介绍了oBERTa语言模型系列，这是一组易于使用的语言模型，可使自然语言处理（NLP）从业者在无需模型压缩专业知识的情况下获得3.8至24.3倍的模型加速效果。具体而言，oBERTa扩展了现有关于剪枝、知识蒸馏和量化的工作，通过利用冻结嵌入改进知识蒸馏，并优化模型初始化方法，从而在广泛的迁移任务中实现更高精度。在构建oBERTa的过程中，我们探究了高度优化的RoBERTa与BERT在预训练和微调阶段剪枝行为上的差异，发现RoBERTa在微调阶段对压缩的适应性较弱。我们在七个具有代表性的NLP任务上验证了oBERTa的性能，结果表明：改进的压缩技术使剪枝后的oBERTa模型在SQUAD V1.1问答数据集上能达到BERTBASE的精度水平，并超越Prune OFA Large模型，尽管其推理速度分别快8倍和2倍。我们公开了代码、训练方案及相关模型，以促进广泛应用与实验研究。