We consider a K-armed bandit problem in general graphs where agents are arbitrarily connected and each of them has limited memorizing capabilities and communication bandwidth. The goal is to let each of the agents eventually learn the best arm. It is assumed in these studies that the communication graph should be complete or well-structured, whereas such an assumption is not always valid in practice. Furthermore, limited memorization and communication bandwidth also restrict the collaborations of the agents, since the agents memorize and communicate very few experiences. Additionally, an agent may be corrupted to share falsified experiences to its peers, while the resource limit in terms of memorization and communication may considerably restrict the reliability of the learning process. To address the above issues, we propose a three-staged collaborative learning algorithm. In each step, the agents share their latest experiences with each other through light-weight random walks in a general communication graph, and then make decisions on which arms to pull according to the recommendations received from their peers. The agents finally update their adoptions (i.e., preferences to the arms) based on the reward obtained by pulling the arms. Our theoretical analysis shows that, when there are a sufficient number of agents participating in the collaborative learning process, all the agents eventually learn the best arm with high probability, even with limited memorizing capabilities and light-weight communications. We also reveal in our theoretical analysis the upper bound on the number of corrupted agents our algorithm can tolerate. The efficacy of our proposed three-staged collaborative learning algorithm is finally verified by extensive experiments on both synthetic and real datasets.

翻译：我们考虑一般图中的K臂赌博机问题，其中智能体任意连接且每个智能体具有有限的记忆能力和通信带宽。目标是让每个智能体最终学会最优臂。现有研究假设通信图必须是完全图或结构良好的，但这一假设在实践中往往不成立。此外，有限的记忆和通信带宽也限制了智能体的协作，因为智能体记忆和共享的经验非常少。进一步而言，智能体可能被恶意篡改而向同伴分享虚假经验，而记忆和通信方面的资源限制会显著降低学习过程的可靠性。为解决上述问题，我们提出一种三阶段协作学习算法。在每一步中，智能体通过轻量级随机游走在一般通信图中相互共享最新经验，然后根据从同伴处收到的推荐决定拉取哪个臂。最终，智能体根据拉取臂获得的奖励更新其采纳度（即对臂的偏好）。我们的理论分析表明，当参与协作学习过程的智能体数量足够多时，即使记忆能力和通信能力有限，所有智能体也能以高概率最终学会最优臂。我们还通过理论分析揭示了算法能够容忍的恶意智能体数量上限。最后，在合成数据集和真实数据集上的大量实验验证了我们提出的三阶段协作学习算法的有效性。