Large language models show impressive results at predicting structured text such as code, but also commonly introduce errors and hallucinations in their output. When used to assist software developers, these models may make mistakes that users must go back and fix, or worse, introduce subtle bugs that users may miss entirely. We propose Randomized Utility-driven Synthesis of Uncertain REgions (R-U-SURE), an approach for building uncertainty-aware suggestions based on a decision-theoretic model of goal-conditioned utility, using random samples from a generative model as a proxy for the unobserved possible intents of the end user. Our technique combines minimum-Bayes-risk decoding, dual decomposition, and decision diagrams in order to efficiently produce structured uncertainty summaries, given only sample access to an arbitrary generative model of code and an optional AST parser. We demonstrate R-U-SURE on three developer-assistance tasks, and show that it can be applied different user interaction patterns without retraining the model and leads to more accurate uncertainty estimates than token-probability baselines.

翻译：大语言模型在预测代码等结构化文本方面展现出显著成效，但其输出中常引入错误与幻觉。当它们被用于辅助软件开发人员时，这些模型可能产生用户必须事后修正的错误，甚至更糟——引入用户可能完全忽略的隐蔽缺陷。本文提出随机意图驱动的不确定区域综合方法（R-U-SURE），这是一种基于目标条件效用决策理论模型构建不确定性感知建议的方法，通过将生成模型的随机样本作为终端用户未观测可能意图的代理。我们的技术结合最小贝叶斯风险解码、对偶分解与决策图，仅需访问任意代码生成模型的样本接口及可选的抽象语法树解析器，即可高效生成结构化不确定性摘要。我们在三个开发者辅助任务上验证了R-U-SURE，证明其无需重新训练模型即可适配不同用户交互模式，并且相较于基于token概率的基线方法，能够产生更准确的不确定性估计。