We investigate the role of uncertainty in decision-making problems with natural language as input. For such tasks, using Large Language Models as agents has become the norm. However, none of the recent approaches employ any additional phase for estimating the uncertainty the agent has about the world during the decision-making task. We focus on a fundamental decision-making framework with natural language as input, which is the one of contextual bandits, where the context information consists of text. As a representative of the approaches with no uncertainty estimation, we consider an LLM bandit with a greedy policy, which picks the action corresponding to the largest predicted reward. We compare this baseline to LLM bandits that make active use of uncertainty estimation by integrating the uncertainty in a Thompson Sampling policy. We employ different techniques for uncertainty estimation, such as Laplace Approximation, Dropout, and Epinets. We empirically show on real-world data that the greedy policy performs worse than the Thompson Sampling policies. These findings suggest that, while overlooked in the LLM literature, uncertainty plays a fundamental role in bandit tasks with LLMs.

翻译：我们研究了以自然语言为输入的决策问题中不确定性的作用。在此类任务中，使用大语言模型作为智能体已成为常态。然而，近期方法均未在决策过程中额外引入对智能体关于世界认知不确定性的估计阶段。我们聚焦于以自然语言为输入的基础决策框架——上下文赌博机模型，其中上下文信息由文本构成。作为无不确定性估计方法的代表，我们考虑了采用贪心策略的LLM赌博机，该策略选择对应最大预测奖励的动作。我们将此基准与通过将不确定性整合到汤普森采样策略中主动利用不确定性估计的LLM赌博机进行比较。我们采用了多种不确定性估计技术，包括拉普拉斯近似、Dropout和Epinets。基于真实数据的实验表明，贪心策略的表现逊于汤普森采样策略。这些发现表明，尽管在LLM文献中常被忽视，不确定性在基于LLM的赌博机任务中扮演着基础性角色。