Safe deployment of large language models (LLMs) may benefit from a reliable method for assessing their generated content to determine when to abstain or to selectively generate. While likelihood-based metrics such as perplexity are widely employed, recent research has demonstrated the limitations of using sequence-level probability estimates given by LLMs as reliable indicators of generation quality. Conversely, LLMs have demonstrated strong calibration at the token level, particularly when it comes to choosing correct answers in multiple-choice questions or evaluating true/false statements. In this work, we reformulate open-ended generation tasks into token-level prediction tasks, and leverage LLMs' superior calibration at the token level. We instruct an LLM to self-evaluate its answers, employing either a multi-way comparison or a point-wise evaluation approach, with the option to include a ``None of the above'' option to express the model's uncertainty explicitly. We benchmark a range of scoring methods based on self-evaluation and evaluate their performance in selective generation using TruthfulQA and TL;DR. Through experiments with PaLM-2 and GPT-3, we demonstrate that self-evaluation based scores not only improve accuracy, but also correlate better with the overall quality of generated content.

翻译：大语言模型（LLM）的安全部署可能需要一种可靠的方法来评估其生成内容，以决定何时应弃权或选择性生成。尽管基于似然度的指标（如困惑度）被广泛使用，但近期研究表明，LLM给出的序列级概率估计作为生成质量的可靠指标存在局限性。相比之下，LLM在词元级展现出良好的校准能力，尤其在多项选择题中选择正确答案或评估真/假陈述时表现突出。本研究将开放式生成任务重新表述为词元级预测任务，并利用LLM在词元级的优越校准特性。我们指引LLM对其答案进行自我评估，采用多路比较或逐点评估方法，并可选择引入"以上皆非"选项以明确表达模型的不确定性。我们基于自我评估方法系统评估了一系列评分方案，并在TruthfulQA和TL;DR数据集上测试其在选择性生成中的表现。通过PaLM-2和GPT-3的实验表明，基于自我评估的评分不仅能提升准确性，还能更好地与生成内容的整体质量相关联。