In high-stakes AI applications, even a single action can cause irreparable damage. However, nearly all of sequential decision-making theory assumes that all errors are recoverable (e.g., by bounding rewards). Standard bandit algorithms that explore aggressively may cause irreparable damage when this assumption fails. Some prior work avoids irreparable errors by asking for help from a mentor, but a mentor may not always be available. In this work, we formalize a model of learning with unbounded rewards without a mentor as a two-action contextual bandit with an abstain option: at each round the agent observes an input and chooses either to abstain (always 0 reward) or to commit (execute a preexisting task policy). Committing yields rewards that are upper-bounded but can be arbitrarily negative, and the commit reward is assumed Lipschitz in the input. We propose a caution-based algorithm that learns when not to learn: it chooses a trusted region and commits only where the available evidence does not already certify harm. Under these conditions and i.i.d. inputs, we establish sublinear regret guarantees, theoretically demonstrating the effectiveness of cautious exploration for deploying learning agents safely in high-stakes environments.

翻译：在高风险人工智能应用中，单个行动即可能造成不可逆的损害。然而，几乎所有序列决策理论都默认所有错误皆可挽回（例如通过限制奖励范围）。当这一假设不成立时，标准赌博机算法的激进探索可能导致不可逆损害。先前研究通过向导师求助来避免不可逆错误，但导师未必始终可用。本研究将无导师参与的无界奖励学习形式化为包含弃权选项的双动作上下文赌博机模型：智能体每轮观测输入后，可选择弃权（奖励恒为0）或执行（调用预设任务策略）。执行动作产生的奖励存在上界但可任意负向，且假设执行奖励随输入满足利普希茨连续性。我们提出一种基于谨慎原则的算法，该算法能学习何时应停止学习：它划定可信区域，仅在现有证据无法确证损害时执行动作。在独立同分布输入条件下，我们建立了次线性遗憾保证，从理论上证明了谨慎探索策略在高风险环境中安全部署学习智能体的有效性。