The idea of decision-aware model learning, that models should be accurate where it matters for decision-making, has gained prominence in model-based reinforcement learning. While promising theoretical results have been established, the empirical performance of algorithms leveraging a decision-aware loss has been lacking, especially in continuous control problems. In this paper, we present a study on the necessary components for decision-aware reinforcement learning models and we showcase design choices that enable well-performing algorithms. To this end, we provide a theoretical and empirical investigation into prominent algorithmic ideas in the field. We highlight that empirical design decisions established in the MuZero line of works are vital to achieving good performance for related algorithms, and we showcase differences in behavior between different instantiations of value-aware algorithms in stochastic environments. Using these insights, we propose the Latent Model-Based Decision-Aware Actor-Critic framework ($\lambda$-AC) for decision-aware model-based reinforcement learning in continuous state-spaces and highlight important design choices in different environments.

翻译：决策感知模型学习（即模型应在决策关键区域保持高精度）在基于模型的强化学习中日益受到重视。尽管已建立有前景的理论成果，但利用决策感知损失函数的算法在经验性能上仍显不足，尤其在连续控制问题中。本文系统研究了决策感知强化学习模型的关键组件，并展示了能实现高性能算法的设计选择。为此，我们对该领域主流算法思想进行了理论与实证探究。研究表明，MuZero系列算法中确立的经验设计决策对于相关算法的性能实现至关重要，同时揭示了随机环境中不同价值感知算法实例的行为差异。基于这些发现，我们提出了连续状态空间下决策感知模型强化学习的隐式模型决策感知演员-评论家框架（$\lambda$-AC），并强调了不同环境中的关键设计选择。