The stochastic multi-armed bandit model captures the tradeoff between exploration and exploitation. We study the effects of competition and cooperation on this tradeoff. Suppose there are $k$ arms and two players, Alice and Bob. In every round, each player pulls an arm, receives the resulting reward, and observes the choice of the other player but not their reward. Alice's utility is $\Gamma_A + \lambda \Gamma_B$ (and similarly for Bob), where $\Gamma_A$ is Alice's total reward and $\lambda \in [-1, 1]$ is a cooperation parameter. At $\lambda = -1$ the players are competing in a zero-sum game, at $\lambda = 1$, they are fully cooperating, and at $\lambda = 0$, they are neutral: each player's utility is their own reward. The model is related to the economics literature on strategic experimentation, where usually players observe each other's rewards. With discount factor $\beta$, the Gittins index reduces the one-player problem to the comparison between a risky arm, with a prior $\mu$, and a predictable arm, with success probability $p$. The value of $p$ where the player is indifferent between the arms is the Gittins index $g = g(\mu,\beta) > m$, where $m$ is the mean of the risky arm. We show that competing players explore less than a single player: there is $p^* \in (m, g)$ so that for all $p > p^*$, the players stay at the predictable arm. However, the players are not myopic: they still explore for some $p > m$. On the other hand, cooperating players explore more than a single player. We also show that neutral players learn from each other, receiving strictly higher total rewards than they would playing alone, for all $ p\in (p^*, g)$, where $p^*$ is the threshold from the competing case. Finally, we show that competing and neutral players eventually settle on the same arm in every Nash equilibrium, while this can fail for cooperating players.

翻译：随机多臂赌博机模型刻画了探索与利用之间的权衡。我们研究了竞争与合作对此权衡的影响。假设有$k$个臂和两名玩家Alice与Bob。每轮中，每位玩家拉动一个臂并获得相应奖励，同时观察到对方的选择但非其奖励。Alice的效用为$\Gamma_A + \lambda \Gamma_B$（Bob同理），其中$\Gamma_A$为Alice的总奖励，$\lambda \in [-1, 1]$是合作参数。当$\lambda = -1$时，玩家在零和博弈中竞争；$\lambda = 1$时，玩家完全合作；$\lambda = 0$时，玩家保持中立：每位玩家的效用即自身奖励。该模型与战略实验的经济学文献相关，通常这类文献中玩家会互相观察奖励。在折扣因子$\beta$下，Gittins指数将单玩家问题简化为对先验均值$\mu$的风险臂与成功概率$p$的可预测臂进行比较。令玩家在两臂间无差异的$p$值即为Gittins指数$g = g(\mu,\beta) > m$，其中$m$为风险臂的均值。我们证明：竞争型玩家的探索少于单人情形——存在阈值$p^* \in (m, g)$，使得对所有$p > p^*$，玩家会始终停留在可预测臂。然而玩家并非短视：对于某些$p > m$，他们仍会进行探索。另一方面，合作型玩家的探索多于单人情形。我们还证明：中立型玩家会互相学习——当$p \in (p^*, g)$时（$p^*$为竞争情形的阈值），他们获得的总奖励严格高于独自博弈。最后，我们证明：在每个纳什均衡中，竞争型和中立型玩家最终会收敛至同一臂，而合作型玩家则未必如此。