In a sequential decision-making problem, the information structure is the description of how events in the system occurring at different points in time affect each other. Classical models of reinforcement learning (e.g., MDPs, POMDPs, Dec-POMDPs, and POMGs) assume a very simple and highly regular information structure, while more general models like predictive state representations do not explicitly model the information structure. By contrast, real-world sequential decision-making problems typically involve a complex and time-varying interdependence of system variables, requiring a rich and flexible representation of information structure. In this paper, we argue for the perspective that explicit representation of information structures is an important component of analyzing and solving reinforcement learning problems. We propose novel reinforcement learning models with an explicit representation of information structure, capturing classical models as special cases. We show that this leads to a richer analysis of sequential decision-making problems and enables more tailored algorithm design. In particular, we characterize the "complexity" of the observable dynamics of any sequential decision-making problem through a graph-theoretic analysis of the DAG representation of its information structure. The central quantity in this analysis is the minimal set of variables that $d$-separates the past observations from future observations. Furthermore, through constructing a generalization of predictive state representations, we propose tailored reinforcement learning algorithms and prove that the sample complexity is in part determined by the information structure. This recovers known tractability results and gives a novel perspective on reinforcement learning in general sequential decision-making problems, providing a systematic way of identifying new tractable classes of problems.

翻译：在序列决策问题中，信息结构描述了系统中不同时间点发生的事件如何相互影响。经典的强化学习模型（如MDP、POMDP、Dec-POMDP和POMG）假设了极其简单且高度规则的信息结构，而预测状态表征等更通用的模型并未显式建模信息结构。相比之下，现实世界的序列决策问题通常涉及系统变量间复杂且时变的相互依赖关系，需要丰富灵活的信息结构表征。本文主张将信息结构的显式表征作为分析与求解强化学习问题的重要组成部分。我们提出了具有显式信息结构表征的新型强化学习模型，将经典模型作为特例纳入其中。研究表明，该方法能够对序列决策问题进行更深入的分析，并实现更具针对性的算法设计。具体而言，通过对其信息结构的有向无环图（DAG）表征进行图论分析，我们刻画了任意序列决策问题可观测动力学的"复杂性"。该分析的核心量是能够实现过去观测与未来观测d-分离的最小变量集。此外，通过构建预测状态表征的泛化形式，我们提出了定制化的强化学习算法，并证明其样本复杂度部分由信息结构决定。这一成果既复现了已知的可解性结论，又为通用序列决策问题中的强化学习提供了全新视角，系统性地识别出新的可解问题类别。