Reinforcement learning (RL) for robot control typically requires a detailed representation of the environment state, including information about task-relevant objects not directly measurable. Keypoint detectors, such as spatial autoencoders (SAEs), are a common approach to extracting a low-dimensional representation from high-dimensional image data. SAEs aim at spatial features such as object positions, which are often useful representations in robotic RL. However, whether an SAE is actually able to track objects in the scene and thus yields a spatial state representation well suited for RL tasks has rarely been examined due to a lack of established metrics. In this paper, we propose to assess the performance of an SAE instance by measuring how well keypoints track ground truth objects in images. We present a computationally lightweight metric and use it to evaluate common baseline SAE architectures on image data from a simulated robot task. We find that common SAEs differ substantially in their spatial extraction capability. Furthermore, we validate that SAEs that perform well in our metric achieve superior performance when used in downstream RL. Thus, our metric is an effective and lightweight indicator of RL performance before executing expensive RL training. Building on these insights, we identify three key modifications of SAE architectures to improve tracking performance. We make our code available at anonymous.4open.science/r/sae-rl.

翻译：强化学习（RL）在机器人控制中通常需要详细的环境状态表征，包括与任务相关但无法直接测量的物体信息。关键点检测器（如空间自编码器SAE）是从高维图像数据中提取低维表征的常用方法。SAE旨在提取目标位置等空间特征，这些特征在机器人强化学习中常被用作有效表征。然而，由于缺乏公认的度量标准，SAE是否真正能够追踪场景中的物体，从而生成适合RL任务的空间状态表征，这一点鲜有评估。本文提出通过衡量关键点对图像中真实物体的追踪精度来评估SAE模型性能。我们提出一种计算轻量化的度量指标，并利用它评估模拟机器人任务图像数据上的常见SAE基线架构。研究发现，不同SAE的空间提取能力存在显著差异。此外，我们验证了在该度量指标中表现优异的SAE在下游RL任务中能取得更优性能。因此，该指标可在大规模RL训练执行前作为有效的轻量化性能预判工具。基于这些发现，我们提出三种改进SAE架构的关键方法以提升追踪性能。代码已开源至anonymous.4open.science/r/sae-rl。