Ensuring that deep learning models are well-calibrated in terms of their predictive uncertainty is essential in maintaining their trustworthiness and reliability, yet despite increasing advances in foundation model research, the relationship between such large language models (LLMs) and their calibration remains an open area of research. In this work, we look at a critical gap in the calibration of LLMs within multilingual settings, in an attempt to better understand how the data scarcity can potentially lead to different calibration effects and how commonly used techniques can apply in these settings. Our analysis on two multilingual benchmarks, over 29 and 42 languages respectively, reveals that even in low-resource languages, model confidence can increase significantly after instruction-tuning on high-resource language SFT datasets. However, improvements in accuracy are marginal or non-existent, resulting in mis-calibration, highlighting a critical shortcoming of standard SFT for multilingual languages. Furthermore, we observe that the use of label smoothing to be a reasonable method alleviate this concern, again without any need for low-resource SFT data, maintaining better calibration across all languages. Overall, this highlights the importance of multilingual considerations for both training and tuning LLMs in order to improve their reliability and fairness in downstream use.

翻译：确保深度学习模型在其预测不确定性方面得到良好校准，对于维持其可信度与可靠性至关重要。然而，尽管基础模型研究不断取得进展，大型语言模型（LLMs）与其校准之间的关系仍是一个开放的研究领域。在本工作中，我们着眼于多语言环境下LLMs校准中的一个关键缺口，试图更好地理解数据稀缺如何可能导致不同的校准效应，以及常用技术在这些环境下的适用性。我们在两个多语言基准（分别涵盖29种和42种语言）上的分析表明，即使在低资源语言中，模型在高资源语言监督微调（SFT）数据集上进行指令调优后，其置信度也会显著提升。然而，准确率的改善却微乎其微甚至不存在，从而导致校准失准，这凸显了标准SFT方法在多语言场景下的一个关键缺陷。此外，我们观察到使用标签平滑是一种合理的方法，可在无需任何低资源SFT数据的情况下缓解这一问题，从而在所有语言中保持更好的校准状态。总体而言，这项工作强调了在LLMs的训练与调优过程中进行多语言考量的重要性，以提升其在下游应用中的可靠性与公平性。