One of the fundamental goals of visual perception is to allow agents to meaningfully interact with their environment. In this paper, we take a step towards that long-term goal -- we extract highly localized actionable information related to elementary actions such as pushing or pulling for articulated objects with movable parts. For example, given a drawer, our network predicts that applying a pulling force on the handle opens the drawer. We propose, discuss, and evaluate novel network architectures that given image and depth data, predict the set of actions possible at each pixel, and the regions over articulated parts that are likely to move under the force. We propose a learning-from-interaction framework with an online data sampling strategy that allows us to train the network in simulation (SAPIEN) and generalizes across categories. But more importantly, our learned models even transfer to real-world data. Check the project website for the code and data release.

翻译：视觉感知的基本目标之一是让代理商有意义地与环境互动。在本文中,我们朝着这一长期目标迈出了一步 -- -- 我们提取了与基本行动有关的高度局部化的可操作信息,例如推动或拉动可移动部件的分解物体。例如,从抽屉中,我们的网络预测在抽屉中运用拉动力将抽屉打开。我们提议、讨论和评估具有图像和深度数据的新颖网络结构,预测每像素可能采取的行动,以及每个像素中可能移动的分解部分的区域。我们提议了一个在线数据取样战略,让我们能够对网络进行模拟(SAPIEN)和跨类的培训。但更重要的是,我们所学的模型甚至可以传输到真实世界的数据。我们查看项目网站的代码和数据发布。