Social media platforms prevent malicious activities by detecting harmful content of posts and comments. To that end, they employ large-scale deep neural network language models for sentiment analysis and content understanding. Some models, like BERT, are complex, and have numerous parameters, which makes them expensive to operate and maintain. To overcome these deficiencies, industry experts employ a knowledge distillation compression technique, where a distilled model is trained to reproduce the classification behavior of the original model. The distillation processes terminates when the distillation loss function reaches the stopping criteria. This function is mainly designed to ensure that the original and the distilled models exhibit alike classification behaviors. However, besides classification accuracy, there are additional properties of the original model that the distilled model should preserve to be considered as an appropriate abstraction. In this work, we explore whether distilled TinyBERT models preserve confidence values of the original BERT models, and investigate how this confidence preservation property could guide tuning hyperparameters of the distillation process.

翻译：社交媒体平台通过检测帖子和评论中的有害内容来防止恶意行为。为此，它们采用大规模深度神经网络语言模型进行情感分析和内容理解。像BERT这样的模型结构复杂且参数众多，导致其运行和维护成本高昂。为克服这些不足，行业专家采用知识蒸馏压缩技术，训练一个蒸馏模型来复现原始模型的分类行为。当蒸馏损失函数达到停止准则时，蒸馏过程终止。该函数主要用于确保原始模型与蒸馏模型表现出相似的分类行为。然而，除分类准确性外，蒸馏模型还应保留原始模型的其他属性，才能被视为合格的抽象。本文探究了蒸馏后的TinyBERT模型是否保留了原始BERT模型的置信度值，并研究了这种置信度保持特性如何指导蒸馏过程的超参数调优。