We consider the problem of learning personalized decision policies on observational data from heterogeneous data sources. Moreover, we examine this problem in the federated setting where a central server aims to learn a policy on the data distributed across the heterogeneous sources without exchanging their raw data. We present a federated policy learning algorithm based on aggregation of local policies trained with doubly robust offline policy evaluation and learning strategies. We provide a novel regret analysis for our approach that establishes a finite-sample upper bound on a notion of global regret across a distribution of clients. In addition, for any individual client, we establish a corresponding local regret upper bound characterized by the presence of distribution shift relative to all other clients. We support our theoretical findings with experimental results. Our analysis and experiments provide insights into the value of heterogeneous client participation in federation for policy learning in heterogeneous settings.

翻译：本文研究利用异质数据源中的观测数据学习个性化决策策略的问题。我们在联邦设置下审视该问题，即中央服务器旨在基于分布在不同异质数据源上的数据学习策略，而不交换原始数据。我们提出了一种基于局部策略聚合的联邦策略学习算法，这些局部策略采用双重鲁棒离线策略评估与学习策略进行训练。我们为该算法提供了一种新颖的遗憾分析，该分析在客户端分布上建立了全局遗憾概念的有限样本上界。此外，对于任意单个客户端，我们建立了由所有其他客户端相对于该客户端的分布偏移所刻画对应的局部遗憾上界。我们通过实验结果支持了理论发现。我们的分析与实验为异质客户端参与联邦学习在异质环境策略学习中的价值提供了见解。