In this work, we study a natural nonparametric estimator of the transition probability matrices of a finite controlled Markov chain. We consider an offline setting with a fixed dataset, collected using a so-called logging policy. We develop sample complexity bounds for the estimator and establish conditions for minimaxity. Our statistical bounds depend on the logging policy through its mixing properties. We show that achieving a particular statistical risk bound involves a subtle and interesting trade-off between the strength of the mixing properties and the number of samples. We demonstrate the validity of our results under various examples, such as ergodic Markov chains, weakly ergodic inhomogeneous Markov chains, and controlled Markov chains with non-stationary Markov, episodic, and greedy controls. Lastly, we use these sample complexity bounds to establish concomitant ones for offline evaluation of stationary Markov control policies.

翻译：本文研究有限受控马尔可夫链转移概率矩阵的一种自然非参数估计器。我们考虑使用所谓记录策略收集的固定数据集的离线设置。我们为该估计器建立了样本复杂度界，并确定了实现极小极大性的条件。我们的统计界依赖于记录策略的混合特性。研究表明，达到特定的统计风险界涉及混合强度与样本数量之间微妙而有趣的权衡。我们在多种示例下验证了结果的有效性，包括遍历马尔可夫链、弱遍历非齐次马尔可夫链，以及具有非平稳马尔可夫控制、情节控制和贪婪控制的受控马尔可夫链。最后，我们利用这些样本复杂度界为平稳马尔可夫控制策略的离线评估建立了相应的界。