Limited computational budgets often prevent transformers from being used in production and from having their high accuracy utilized. A knowledge distillation approach addresses the computational efficiency by self-distilling BERT into a smaller transformer representation having fewer layers and smaller internal embedding. However, the performance of these models drops as we reduce the number of layers, notably in advanced NLP tasks such as span question answering. In addition, a separate model must be trained for each inference scenario with its distinct computational budget. Dynamic-TinyBERT tackles both limitations by partially implementing the Length Adaptive Transformer (LAT) technique onto TinyBERT, achieving x3 speedup over BERT-base with minimal accuracy loss. In this work, we expand the Dynamic-TinyBERT approach to generate a much more highly efficient model. We use MiniLM distillation jointly with the LAT method, and we further enhance the efficiency by applying low-bit quantization. Our quantized length-adaptive MiniLM model (QuaLA-MiniLM) is trained only once, dynamically fits any inference scenario, and achieves an accuracy-efficiency trade-off superior to any other efficient approaches per any computational budget on the SQuAD1.1 dataset (up to x8.8 speedup with <1% accuracy loss). The code to reproduce this work is publicly available on Github.

翻译：有限的计算预算常常使得Transformer无法投入生产应用，也无法充分发挥其高精度优势。知识蒸馏方法通过将BERT自蒸馏为层数更少、内部嵌入更小的轻量级Transformer表示，提升了计算效率。然而，随着层数减少，这些模型的性能会下降，尤其是在跨度问答等高级NLP任务中。此外，每个具有不同计算预算的推理场景都需要单独训练一个模型。Dynamic-TinyBERT通过将长度自适应Transformer（LAT）技术部分应用于TinyBERT，同时解决了这两个限制，相比BERT-base实现了3倍加速且精度损失极小。在本工作中，我们扩展了Dynamic-TinyBERT方法，生成了一个更高效的模型。我们将MiniLM蒸馏与LAT方法相结合，并通过应用低位量化进一步提升了效率。我们的量化长度自适应MiniLM模型（QuaLA-MiniLM）仅需训练一次，即可动态适配任意推理场景，并且在SQuAD1.1数据集上，针对任意计算预算，其精度-效率权衡均优于其他高效方法（最高实现8.8倍加速，精度损失小于1%）。本工作的复现代码已公开在Github上。