We present Unified Latent Dynamics (ULD), a novel reinforcement learning algorithm that unifies the efficiency of model-free methods with the representational strengths of model-based approaches, without incurring planning overhead. By embedding state-action pairs into a latent space in which the true value function is approximately linear, our method supports a single set of hyperparameters across diverse domains -- from continuous control with low-dimensional and pixel inputs to high-dimensional Atari games. We prove that, under mild conditions, the fixed point of our embedding-based temporal-difference updates coincides with that of a corresponding linear model-based value expansion, and we derive explicit error bounds relating embedding fidelity to value approximation quality. In practice, ULD employs synchronized updates of encoder, value, and policy networks, auxiliary losses for short-horizon predictive dynamics, and reward-scale normalization to ensure stable learning under sparse rewards. Evaluated on 80 environments spanning Gym locomotion, DeepMind Control (proprioceptive and visual), and Atari, our approach matches or exceeds the performance of specialized model-free and general model-based baselines -- achieving cross-domain competence with minimal tuning and a fraction of the parameter footprint. These results indicate that value-aligned latent representations alone can deliver the adaptability and sample efficiency traditionally attributed to full model-based planning.

翻译：我们提出统一潜在动力学（ULD），一种新颖的强化学习算法，它统一了无模型方法的效率与基于模型方法的表示优势，且不引入规划开销。通过将状态-动作对嵌入到真实价值函数近似线性的潜在空间中，我们的方法支持跨不同领域使用单一超参数集——从具有低维和像素输入的连续控制到高维Atari游戏。我们证明，在温和条件下，基于嵌入的时间差分更新的不动点与对应线性基于模型价值展开的不动点重合，并推导出将嵌入保真度与价值近似质量相关联的显式误差界。在实践中，ULD采用编码器、价值网络与策略网络的同步更新，结合短时域预测动力学的辅助损失以及奖励尺度归一化，以确保稀疏奖励下的稳定学习。在涵盖Gym运动控制、DeepMind Control（本体感知与视觉）及Atari的80个环境中评估，我们的方法匹配或超越了专用无模型与通用基于模型基线的性能——以最小调参需求和少量参数占用实现了跨领域能力。这些结果表明，仅凭价值对齐的潜在表示即可提供传统上归因于完整基于模型规划的适应性与样本效率。