Reinforcement Learning (RL), bolstered by the expressive capabilities of Deep Neural Networks (DNNs) for function approximation, has demonstrated considerable success in numerous applications. However, its practicality in addressing various real-world scenarios, characterized by diverse and unpredictable dynamics, noisy signals, and large state and action spaces, remains limited. This limitation stems from issues such as poor data efficiency, limited generalization capabilities, a lack of safety guarantees, and the absence of interpretability, among other factors. To overcome these challenges and improve performance across these crucial metrics, one promising avenue is to incorporate additional structural information about the problem into the RL learning process. Various sub-fields of RL have proposed methods for incorporating such inductive biases. We amalgamate these diverse methodologies under a unified framework, shedding light on the role of structure in the learning problem, and classify these methods into distinct patterns of incorporating structure. By leveraging this comprehensive framework, we provide valuable insights into the challenges of structured RL and lay the groundwork for a design pattern perspective on RL research. This novel perspective paves the way for future advancements and aids in developing more effective and efficient RL algorithms that can potentially handle real-world scenarios better.

翻译：强化学习（Reinforcement Learning, RL）借助深度神经网络（Deep Neural Networks, DNNs）强大的函数逼近能力，已在众多应用中展现出显著成功。然而，其在处理具有多样且不可预测的动力学、噪声信号以及大规模状态和动作空间的各类真实场景时，实用性仍然有限。这种局限性源于数据效率低下、泛化能力受限、缺乏安全保障和可解释性缺失等问题。为克服这些挑战并提升在这些关键指标上的性能，一个颇具前景的途径是将问题的额外结构信息融入RL学习过程。RL的多个子领域已提出纳入此类归纳偏置的方法。我们将这些多样化的方法整合到统一框架下，阐明结构在学习问题中的作用，并将这些方法分类为引入结构的不同模式。通过利用这一全面框架，我们为结构化RL所面临的挑战提供了宝贵见解，并为RL研究中设计模式视角的建立奠定了基础。这一新颖视角为未来发展铺平道路，有助于开发出更有效、更高效的RL算法，从而能更好地应对真实场景。