Fine-tuned Large Language Models (LLMs) often suffer from overconfidence and poor calibration, particularly when fine-tuned on small datasets. To address these challenges, we propose a simple combination of Low-Rank Adaptation (LoRA) with Gaussian Stochastic Weight Averaging (SWAG), facilitating approximate Bayesian inference in LLMs. Through extensive testing across several Natural Language Processing (NLP) benchmarks, we demonstrate that our straightforward and computationally efficient approach improves model generalization and calibration. We further show that our method exhibits greater robustness against distribution shift, as reflected in its performance on out-of-distribution tasks.

翻译：经微调的大型语言模型（LLMs）常面临过度自信和校准不良的问题，尤其是在小数据集上微调时更为突出。为解决这些挑战，我们提出了一种将低秩适配（LoRA）与高斯随机权重平均（SWAG）相结合的简洁方法，从而在大语言模型中实现近似贝叶斯推理。通过在多个自然语言处理（NLP）基准上的广泛测试，我们证明了这种直接且计算高效的方法能够提升模型的泛化能力和校准性能。我们进一步表明，该方法在分布偏移下表现出更强的鲁棒性，这一点反映在其在分布外任务上的性能中。