Fine-tuning large models is highly effective, however, inference can be expensive and produces carbon emissions. Knowledge distillation has been shown to be a practical solution to reduce inference costs, but the distillation process itself requires significant computational resources. Rather than buying or renting GPUs to fine-tune, then distill a large model, an NLP practitioner might instead choose to allocate the available budget to hire annotators and manually label additional fine-tuning data. In this paper, we investigate how to most efficiently use a fixed budget to build a compact model. Through extensive experiments on six diverse tasks, we show that distilling from T5-XXL (11B) to T5-Small (60M) is almost always a cost-efficient strategy compared to annotating more data to directly train a compact model (T5-Small). We further investigate how the optimal budget allocated towards computation varies across scenarios. We will make our code, datasets, annotation cost estimates, and baseline models available as a benchmark to support further work on cost-efficient training of compact models.

翻译：微调大型模型效果显著，但推理过程成本高昂并产生碳排放。知识蒸馏已被证明是降低推理成本的实用方案，然而蒸馏过程本身需要大量计算资源。自然语言处理从业者在面对有限预算时，可能选择不购买或租用GPU来微调再蒸馏大型模型，而是将可用预算用于雇佣标注者手动标注额外的微调数据。本文研究如何利用固定预算最高效地构建紧凑模型。通过六类多样化任务的广泛实验表明，与标注更多数据直接训练紧凑模型（T5-Small）相比，从T5-XXL（11B）蒸馏至T5-Small（60M）几乎始终是更具成本效益的策略。我们进一步探究了不同场景下计算资源的预算分配最优策略。我们将提供代码、数据集、标注成本估算及基线模型作为基准，以支持紧凑模型经济高效训练的相关研究。