How do you incentivize self-interested agents to $\textit{explore}$ when they prefer to $\textit{exploit}$? We consider complex exploration problems, where each agent faces the same (but unknown) MDP. In contrast with traditional formulations of reinforcement learning, agents control the choice of policies, whereas an algorithm can only issue recommendations. However, the algorithm controls the flow of information, and can incentivize the agents to explore via information asymmetry. We design an algorithm which explores all reachable states in the MDP. We achieve provable guarantees similar to those for incentivizing exploration in static, stateless exploration problems studied previously. To the best of our knowledge, this is the first work to consider mechanism design in a stateful, reinforcement learning setting.

翻译：当自利型智能体偏好于利用（exploit）时，如何激励它们进行探索（explore）？我们考虑复杂的探索问题，其中每个智能体面对相同的（但未知的）马尔可夫决策过程（MDP）。与传统的强化学习框架不同，智能体控制策略的选择，而算法仅能提出建议。然而，算法掌控信息流，并可通过信息不对称来激励智能体进行探索。我们设计了一种算法，用于探索MDP中所有可达状态。我们获得了与先前静态无状态探索问题中激励探索相似的可证明保障。据我们所知，这是首个在含状态的强化学习设置中考虑机制设计的工作。