Understanding the behavior of deep reinforcement learning (DRL) agents is crucial for improving their performance and reliability. However, the complexity of their policies often makes them challenging to understand. In this paper, we introduce a new approach for investigating the behavior modes of DRL policies, which involves utilizing dimensionality reduction and trajectory clustering in the latent space of neural networks. Specifically, we use Pairwise Controlled Manifold Approximation Projection (PaCMAP) for dimensionality reduction and TRACLUS for trajectory clustering to analyze the latent space of a DRL policy trained on the Mountain Car control task. Our methodology helps identify diverse behavior patterns and suboptimal choices by the policy, thus allowing for targeted improvements. We demonstrate how our approach, combined with domain knowledge, can enhance a policy's performance in specific regions of the state space.

翻译：理解深度强化学习（DRL）智能体的行为对于提升其性能与可靠性至关重要。然而，策略的复杂性往往使其难以解读。本文提出了一种研究DRL策略行为模式的新方法，该方法利用神经网络潜空间中的降维技术与轨迹聚类。具体而言，我们采用成对受控流形逼近投影（PaCMAP）进行降维，并借助TRACLUS算法对在“山地车”控制任务上训练的DRL策略的潜空间进行轨迹聚类。该方法有助于识别策略的多样化行为模式与次优决策，从而支持针对性改进。我们展示了该方法如何结合领域知识，在状态空间的特定区域提升策略性能。