Recent work on robot manipulation has advanced policy generalization to novel scenarios. However, it is often difficult to characterize how different evaluation settings actually represent generalization from the training distribution of a given policy. To work towards more precise evaluation of generalization in robotics, we propose RADAR, a scalable framework for directly comparing test-time evaluation tasks to policy training data, to determine what form of policy generalization is required. RADAR consists of a two-stage pipeline: first, retrieval using generalist policy embeddings identifies which training examples are relevant for a given evaluation task. Next, vision-language models (VLMs) analyze the evaluation task against the retrieved data, outputting interpretable analysis on how they compare along a variety of axes, and an overall classification of what type of policy generalization is required. Through controlled experiments, we demonstrate that VLMs are effective at analyzing data for generalization, and that our retrieval step effectively identifies examples needed to make accurate classifications with respect to the training data. Furthermore, we scale RADAR to large-scale datasets, where we observe agreement with human-defined benchmark conditions from prior work. We provide demonstrations at radar-analysis.github.io.

翻译：近期关于机器人操作的研究推动了策略向新场景的泛化能力提升。然而，往往难以量化不同评估设置如何真正体现给定策略对训练分布的泛化程度。为促进机器人领域泛化能力的更精确评估，我们提出RADAR——一种可扩展框架，可直接比较测试时评估任务与策略训练数据，以判定所需策略泛化的具体形式。RADAR包含两阶段流水线：首先，利用通用策略嵌入进行检索，识别与特定评估任务相关的训练样本；随后，视觉语言模型（VLM）分析评估任务与检索数据的对比关系，输出沿多个维度比较的可解释分析结果，并对所需策略泛化类型进行整体分类。通过受控实验，我们证明VLM能有效分析数据的泛化特性，且检索步骤可准确识别出基于训练数据进行分类所需的样本。此外，我们将RADAR扩展至大规模数据集，观察到其与先前工作中人类定义的基准条件高度一致。相关演示见radar-analysis.github.io。