Given a natural language instruction and an input scene, our goal is to train a model to output a manipulation program that can be executed by the robot. Prior approaches for this task possess one of the following limitations: (i) rely on hand-coded symbols for concepts limiting generalization beyond those seen during training [1] (ii) infer action sequences from instructions but require dense sub-goal supervision [2] or (iii) lack semantics required for deeper object-centric reasoning inherent in interpreting complex instructions [3]. In contrast, our approach can handle linguistic as well as perceptual variations, end-to-end trainable and requires no intermediate supervision. The proposed model uses symbolic reasoning constructs that operate on a latent neural object-centric representation, allowing for deeper reasoning over the input scene. Central to our approach is a modular structure consisting of a hierarchical instruction parser and an action simulator to learn disentangled action representations. Our experiments on a simulated environment with a 7-DOF manipulator, consisting of instructions with varying number of steps and scenes with different number of objects, demonstrate that our model is robust to such variations and significantly outperforms baselines, particularly in the generalization settings. The code, dataset and experiment videos are available at https://nsrmp.github.io

翻译：给定自然语言指令和输入场景，我们的目标是训练一个模型，输出可由机器人执行的操作程序。先前方法存在以下局限性之一：（i）依赖手工编码的概念符号，限制了训练未见概念的泛化能力[1]；（ii）从指令推断动作序列，但需要密集的子目标监督[2]；或（iii）缺乏解释复杂指令所必需的深层以对象为中心的语义推理能力[3]。相比之下，我们的方法能处理语言及感知变化，支持端到端训练，且无需中间监督。所提模型使用基于潜在神经对象中心表示运行的符号推理结构，从而对输入场景进行更深层推理。该方法的核心是模块化结构，包含分层指令解析器和动作模拟器，用于学习解耦的动作表示。我们在含7自由度机械臂的仿真环境中进行实验，指令步骤数可变且场景中对象数量不同，结果表明模型对此类变化具有鲁棒性，并在泛化设置中显著优于基线方法。代码、数据集及实验视频见 https://nsrmp.github.io