Pre-trained language models (PLMs) serve as backbones for various real-world systems. For high-stake applications, it's equally essential to have reasonable confidence estimations in predictions. While the vanilla confidence scores of PLMs can already be effectively utilized, PLMs consistently become overconfident in their wrong predictions, which is not desirable in practice. Previous work shows that introducing an extra calibration task can mitigate this issue. The basic idea involves acquiring additional data to train models in predicting the confidence of their initial predictions. However, it only demonstrates the feasibility of this kind of method, assuming that there are abundant extra available samples for the introduced calibration task. In this work, we consider the practical scenario that we need to effectively utilize training samples to make PLMs both task-solvers and self-calibrators. Three challenges are presented, including limited training samples, data imbalance, and distribution shifts. We first conduct pilot experiments to quantify various decisive factors in the calibration task. Based on the empirical analysis results, we propose a training algorithm LM-TOAST to tackle the challenges. Experimental results show that LM-TOAST can effectively utilize the training data to make PLMs have reasonable confidence estimations while maintaining the original task performance. Further, we consider three downstream applications, namely selective classification, adversarial defense, and model cascading, to show the practical usefulness of LM-TOAST. The code will be made public at \url{https://github.com/Yangyi-Chen/LM-TOAST}.

翻译：预训练语言模型（PLMs）作为多种现实世界系统的骨干网络。对于高风险应用而言，在预测中提供合理的置信度估计同样至关重要。尽管PLMs的原始置信度分数已可被有效利用，但PLMs在其错误预测上始终表现出过度自信，这在实践中并不理想。先前研究表明，引入额外的校准任务可以缓解这一问题。其基本思想涉及获取额外数据，训练模型预测其初始预测的置信度。然而，这类方法仅证明了可行性，且假设为引入的校准任务存在大量可用的额外样本。在本工作中，我们考虑实际场景：需要有效利用训练样本，使PLMs兼具任务求解与自校准能力。该场景面临三个挑战：有限的训练样本、数据不平衡以及分布偏移。我们首先通过预实验量化校准任务中的多种决定性因素。基于实证分析结果，我们提出训练算法LM-TOAST以应对这些挑战。实验结果表明，LM-TOAST能有效利用训练数据，在保持原任务性能的同时，使PLMs具备合理的置信度估计能力。此外，我们考虑三个下游应用——选择性分类、对抗防御与模型级联，以展示LM-TOAST的实际效用。代码将在\url{https://github.com/Yangyi-Chen/LM-TOAST} 公开。