Confidence calibration is critical for the safe deployment of machine learning models in the real world. However, such issue in vision-language models like CLIP, particularly after fine-tuning, has not been fully addressed. In this work, we demonstrate that existing prompt tuning methods usually lead to a trade-off of calibration between base and new classes: the cross-entropy loss in CoOp causes overconfidence in new classes by increasing textual label divergence, whereas the regularization of KgCoOp maintains the confidence level but results in underconfidence in base classes due to the improved accuracy. Inspired by the observations, we introduce Dynamic Outlier Regularization (DOR) to ensure the confidence calibration on both base and new classes after fine-tuning. In particular, we propose to minimize the feature deviation of novel textual labels (instead of base classes) sampled from a large vocabulary. In effect, DOR prevents the increase in textual divergence for new labels while easing restrictions on base classes. Extensive experiments demonstrate that DOR can enhance the calibration performance of current fine-tuning methods on base and new classes.

翻译：置信度校准对于机器学习模型在现实世界中的安全部署至关重要。然而，在诸如CLIP等视觉语言模型中，尤其是在微调之后，此类问题尚未得到充分解决。在本工作中，我们证明了现有的提示调优方法通常会导致基础类与新类之间的校准权衡：CoOp中的交叉熵损失通过增加文本标签分歧导致对新类的过度自信，而KgCoOp的正则化虽然保持了置信度水平，但由于准确率的提升，却导致基础类的自信不足。受这些观察的启发，我们引入了动态离群值正则化（DOR），以确保微调后基础类和新类上的置信度校准。具体而言，我们提出最小化从大型词汇表中采样的新文本标签（而非基础类）的特征偏差。实际上，DOR防止了新标签文本分歧的增加，同时放松了对基础类的限制。大量实验表明，DOR能够提升当前微调方法在基础类和新类上的校准性能。