Traditional approaches for manipulation planning rely on an explicit geometric model of the environment to formulate a given task as an optimization problem. However, inferring an accurate model from raw sensor input is a hard problem in itself, in particular for articulated objects (e.g., closets, drawers). In this paper, we propose a Neural Field Representation (NFR) of articulated objects that enables manipulation planning directly from images. Specifically, after taking a few pictures of a new articulated object, we can forward simulate its possible movements, and, therefore, use this neural model directly for planning with trajectory optimization. Additionally, this representation can be used for shape reconstruction, semantic segmentation and image rendering, which provides a strong supervision signal during training and generalization. We show that our model, which was trained only on synthetic images, is able to extract a meaningful representation for unseen objects of the same class, both in simulation and with real images. Furthermore, we demonstrate that the representation enables robotic manipulation of an articulated object in the real world directly from images.

翻译：传统操作规划方法依赖于显式的环境几何模型，将给定任务转化为优化问题。然而，从原始传感器输入中推断精确模型本身就是一个难题，尤其是针对铰接物体（例如橱柜、抽屉）。本文提出了一种铰接物体神经场表示（NFR），能够直接从图像实现操作规划。具体而言，在拍摄几幅新铰接物体的图像后，我们可以正向模拟其可能的运动，从而直接利用该神经模型进行轨迹优化规划。此外，该表示可用于形状重建、语义分割和图像渲染，为训练过程和泛化提供强监督信号。我们证明，仅使用合成图像训练的模型能够提取同类别未知物体的有意义表示（在仿真和真实图像中均有效）。更进一步，我们展示了该表示可直接从图像实现真实世界中铰接物体的机器人操作。