Decentralized stochastic control problems are intrinsically difficult to study because of the inapplicability of standard tools from centralized control such as dynamic programming and the resulting computational complexity. In this paper, we address some of these challenges for decentralized stochastic control with Borel spaces under three different but tightly related information structures under a unified theme: the one-step delayed information sharing pattern, the K-step periodic information sharing pattern, and the completely decentralized information structure where no sharing of information occurs. We will show that the one-step delayed and K-step periodic problems can be reduced to a centralized MDP, generalizing prior results which considered finite, linear, or static models, by addressing several measurability questions. The separated nature of policies under both information structures is then established. We then provide sufficient conditions for the transition kernels of both centralized reductions to be weak-Feller, which facilitates rigorous approximation and learning theoretic results. We will then show that for the completely decentralized control problem finite memory local policies are near optimal under a joint conditional mixing condition. This is achieved by obtaining a bound for finite memory policies which goes to zero as memory size increases. We will also provide a performance bound for the K-periodic problem, which results from replacing the full common information by a finite sliding window of information. The latter will depend on the condition of predictor stability in expected total variation, which we will establish. We finally show that under the periodic information sharing pattern, a quantized Q-learning algorithm converges asymptotically towards a near optimal solution. Each of the above, to our knowledge, is a new contribution to the literature.

翻译：分散随机控制问题本质上是难以研究的，因为集中控制中的标准工具（如动态规划）及其带来的计算复杂性在此不适用。本文针对具有Borel空间的分散随机控制问题，在统一框架下处理三种不同但紧密相关的信息结构所带来的挑战：一步延迟信息共享模式、K步周期信息共享模式以及完全不发生信息共享的完全分散信息结构。我们将证明，通过解决若干可测性问题，一步延迟和K步周期问题可约简为集中化MDP，从而推广了先前仅考虑有限、线性或静态模型的研究结果。随后确立两种信息结构下策略的分离特性。我们进一步给出两种集中化约简的转移核满足弱Feller性质的充分条件，这为严格的近似分析和学习理论结果提供了基础。对于完全分散控制问题，我们证明在联合条件混合条件下，有限记忆局部策略具有近似最优性，这是通过获得随记忆容量增加而趋近于零的有限记忆策略误差界实现的。针对K周期问题，我们通过用有限滑动窗口信息替代完整公共信息，给出了相应的性能误差界，该界将取决于我们建立的期望全变差意义下的预测器稳定性条件。最后，我们证明在周期信息共享模式下，量化Q学习算法能渐近收敛至近似最优解。据我们所知，上述每个结论均为该领域文献中的新贡献。