Fine-tuning pre-trained language models on downstream tasks with varying random seeds has been shown to be unstable, especially on small datasets. Many previous studies have investigated this instability and proposed methods to mitigate it. However, most studies only used the standard deviation of performance scores (SD) as their measure, which is a narrow characterization of instability. In this paper, we analyze SD and six other measures quantifying instability at different levels of granularity. Moreover, we propose a systematic framework to evaluate the validity of these measures. Finally, we analyze the consistency and difference between different measures by reassessing existing instability mitigation methods. We hope our results will inform the development of better measurements of fine-tuning instability.

翻译：在具有不同随机种子的下游任务上微调预训练语言模型已被证明是不稳定的，尤其是在小数据集上。许多先前的研究已经探讨了这种不稳定性并提出了缓解方法。然而，大多数研究仅使用性能得分的标准差（SD）作为度量标准，这是一种对不稳定性的狭义刻画。本文分析了SD以及其他六种在不同粒度水平上量化不稳定性的度量指标。此外，我们提出了一个系统性框架来评估这些度量指标的有效性。最后，通过重新评估现有的不稳定性缓解方法，我们分析了不同度量指标之间的一致性与差异。我们期望我们的研究结果能为开发更好的微调不稳定性度量方法提供参考。