A recent trend in Natural Language Processing is the exponential growth in Language Model (LM) size, which prevents research groups without a necessary hardware infrastructure from participating in the development process. This study investigates methods for Knowledge Distillation (KD) to provide efficient alternatives to large-scale models. In this context, KD means extracting information about language encoded in a Neural Network and Lexical Knowledge Databases. We developed two methods to test our hypothesis that efficient architectures can gain knowledge from LMs and extract valuable information from lexical sources. First, we present a technique to learn confident probability distribution for Masked Language Modeling by prediction weighting of multiple teacher networks. Second, we propose a method for Word Sense Disambiguation (WSD) and lexical KD that is general enough to be adapted to many LMs. Our results show that KD with multiple teachers leads to improved training convergence. When using our lexical pre-training method, LM characteristics are not lost, leading to increased performance in Natural Language Understanding (NLU) tasks over the state-of-the-art while adding no parameters. Moreover, the improved semantic understanding of our model increased the task performance beyond WSD and NLU in a real-problem scenario (Plagiarism Detection). This study suggests that sophisticated training methods and network architectures can be superior over scaling trainable parameters. On this basis, we suggest the research area should encourage the development and use of efficient models and rate impacts resulting from growing LM size equally against task performance.

翻译：自然语言处理领域的最新趋势是语言模型规模的指数级增长，这阻碍了缺乏必要硬件基础设施的研究团队参与开发进程。本研究探讨了知识蒸馏方法，旨在为大规模模型提供高效替代方案。在此背景下，知识蒸馏意味着提取编码在神经网络和词汇知识数据库中的语言信息。我们开发了两种方法，以验证高效架构能够从语言模型中获取知识并从词汇来源提取有价值信息的假设。首先，我们提出一种技术，通过多个教师网络的预测加权来学习掩码语言建模的置信概率分布。其次，我们提出了一种适用于词义消歧和词汇知识蒸馏的方法，该方法具有足够的通用性，可适配多种语言模型。我们的结果表明，使用多个教师网络进行知识蒸馏可以改善训练收敛性。当采用我们的词汇预训练方法时，语言模型的特性得以保留，从而在自然语言理解任务上实现超越当前最优水平的性能提升，且无需增加任何参数。此外，我们模型增强的语义理解能力在真实问题场景（抄袭检测）中提升了超出词义消歧和自然语言理解范围的任务性能。本研究表明，精密的训练方法与网络架构可能优于可训练参数的规模扩展。基于此，我们建议该研究领域应鼓励高效模型的开发与使用，并将语言模型规模增长带来的影响与任务性能置于同等评价地位。