The multi-armed bandit(MAB) problem is a simple yet powerful framework that has been extensively studied in the context of decision-making under uncertainty. In many real-world applications, such as robotic applications, selecting an arm corresponds to a physical action that constrains the choices of the next available arms (actions). Motivated by this, we study an extension of MAB called the graph bandit, where an agent travels over a graph to maximize the reward collected from different nodes. The graph defines the agent's freedom in selecting the next available nodes at each step. We assume the graph structure is fully available, but the reward distributions are unknown. Built upon an offline graph-based planning algorithm and the principle of optimism, we design a learning algorithm, G-UCB, that balances long-term exploration-exploitation using the principle of optimism. We show that our proposed algorithm achieves $O(\sqrt{|S|T\log(T)}+D|S|\log T)$ learning regret, where $|S|$ is the number of nodes and $D$ is the diameter of the graph, which matches the theoretical lower bound $\Omega(\sqrt{|S|T})$ up to logarithmic factors. To our knowledge, this result is among the first tight regret bounds in non-episodic, un-discounted learning problems with known deterministic transitions. Numerical experiments confirm that our algorithm outperforms several benchmarks.

翻译：多臂赌博机（MAB）问题是一个简单而强大的框架，已在不确定性决策情境下得到广泛研究。在机器人等许多实际应用中，选择臂对应着一个物理动作，该动作会限制下一个可用臂（动作）的选择。受此启发，我们研究了MAB的一种扩展形式——图赌博机：智能体在图上移动以最大化从不同节点收集的奖励。图定义了智能体在每一步选择下一可用节点的自由度。我们假设图结构完全已知，但奖励分布未知。基于离线图规划算法和乐观原则，我们设计了学习算法G-UCB，通过乐观原则平衡长期探索与利用。我们证明了所提出算法实现了$O(\sqrt{|S|T\log(T)}+D|S|\log T)$的学习遗憾值，其中$|S|$为节点数，$D$为图的直径，该结果与理论下界$\Omega(\sqrt{|S|T})$仅相差对数因子。据我们所知，此结果是已知确定性转移的非回合制无折扣学习问题中首批紧致遗憾界之一。数值实验证实我们的算法优于多个基准方法。