The idea of decision-aware model learning, that models should be accurate where it matters for decision-making, has gained prominence in model-based reinforcement learning. While promising theoretical results have been established, the empirical performance of algorithms leveraging a decision-aware loss has been lacking, especially in continuous control problems. In this paper, we present a study on the necessary components for decision-aware reinforcement learning models and we showcase design choices that enable well-performing algorithms. To this end, we provide a theoretical and empirical investigation into prominent algorithmic ideas in the field. We highlight that empirical design decisions established in the MuZero line of works are vital to achieving good performance for related algorithms, and we showcase differences in behavior between different instantiations of value-aware algorithms in stochastic environments. Using these insights, we propose the Latent Model-Based Decision-Aware Actor-Critic framework ($\lambda$-AC) for decision-aware model-based reinforcement learning in continuous state-spaces and highlight important design choices in different environments.

翻译：决策感知模型学习理念（即模型应在对决策至关重要的区域保持精确）已在基于模型的强化学习中占据重要地位。尽管已取得具有前景的理论成果，但利用决策感知损失函数的算法在经验性能上仍显不足，尤其在连续控制问题中。本文系统研究了决策感知强化学习模型的必要组件，并展示了能够实现高性能算法的设计选择。为此，我们对该领域的代表性算法思想进行了理论与实证研究。研究表明，MuZero系列工作中确立的经验性设计决策对相关算法取得优异性能至关重要，同时揭示了随机环境中不同价值感知算法实例的行为差异。基于这些见解，我们提出了面向连续状态空间的潜在模型驱动决策感知演员-评论家框架（λ-AC），并针对不同环境重点阐述了关键设计选择。