We study performance-driven environment abstraction for decision-making in large Markov decision processes. Rather than preserving geometric or topological structure, we seek abstractions that directly optimize decision quality. We model abstraction as a controlled approximation obtained by aggregating the state space and enforcing a shared action distribution within each aggregated state. For a fixed partition, we establish a performance guarantee that separates value-function approximation error from the loss introduced by action sharing. Guided by this analysis, we develop a multi-timescale reinforcement learning framework that jointly adapts the policy and a tree-structured environment abstraction. The resulting algorithm refines and coarsens regions of the state space based on Q-value discrepancies, balancing performance against abstraction size and complexity. Empirical results demonstrate substantial state compression, improved sample efficiency, and faster replanning compared to actor-critic baselines.

翻译：我们研究面向性能驱动的环境抽象方法，用于大规模马尔可夫决策过程中的决策制定。不同于保留几何或拓扑结构，我们寻求直接优化决策质量的抽象形式。我们将抽象建模为受控近似，通过聚合状态空间并在每个聚合状态内强制执行共享动作分布实现。针对固定划分，我们建立了性能保证，该保证将价值函数近似误差与动作共享引入的损失分离开来。基于这一分析，我们开发了一个多时间尺度强化学习框架，该框架联合调整策略与树结构环境抽象。所提出的算法根据Q值差异细化或粗化状态空间区域，在性能与抽象规模及复杂度之间取得平衡。实验结果表明，与Actor-Critic基线方法相比，该方法实现了显著的状态压缩、样本效率提升及更快的重规划速度。