Large language models (LLMs) have recently shown great advances in a variety of tasks, including natural language understanding and generation. However, their use in high-stakes decision-making scenarios is still limited due to the potential for errors. Selective prediction is a technique that can be used to improve the reliability of the LLMs by allowing them to abstain from making predictions when they are unsure of the answer. In this work, we propose a novel framework for adaptation with self-evaluation to improve the selective prediction performance of LLMs. Our framework is based on the idea of using parameter-efficient tuning to adapt the LLM to the specific task at hand while improving its ability to perform self-evaluation. We evaluate our method on a variety of question-answering (QA) datasets and show that it outperforms state-of-the-art selective prediction methods. For example, on the CoQA benchmark, our method improves the AUACC from 91.23% to 92.63% and improves the AUROC from 74.61% to 80.25%.

翻译：大语言模型（LLMs）近年来在自然语言理解与生成等多项任务中取得了显著进展。然而，由于存在潜在错误风险，其在高风险决策场景中的应用仍受到限制。选择性预测是一种通过允许模型在不确定答案时放弃预测来提升其可靠性的技术。本文提出了一种新颖的基于自评估的自适应框架，旨在改善大语言模型的选择性预测性能。该框架的核心思想是通过参数高效微调使模型适应特定任务，同时增强其自评估能力。我们在多个问答（QA）数据集上评估了该方法，结果表明其性能优于当前最先进的选择性预测方法。例如，在CoQA基准测试中，我们的方法将AUACC从91.23%提升至92.63%，并将AUROC从74.61%提升至80.25%。