Inspired by the necessity of morphological adaptation in animals, a growing body of work has attempted to expand robot training to encompass physical aspects of a robot's design. However, reinforcement learning methods capable of optimizing the 3D morphology of a robot have been restricted to reorienting or resizing the limbs of a predetermined and static topological genus. Here we show policy gradients for designing freeform robots with arbitrary external and internal structure. This is achieved through actions that deposit or remove bundles of atomic building blocks to form higher-level nonparametric macrostructures such as appendages, organs and cavities. Although results are provided for open loop control only, we discuss how this method could be adapted for closed loop control and sim2real transfer to physical machines in future.

翻译：受动物形态适应必要性的启发，越来越多的研究尝试将机器人训练扩展至涵盖设计的物理层面。然而，能够优化机器人三维形态的强化学习方法此前仅局限于对预设静态拓扑类型的肢体进行方向调整或尺寸缩放。本文展示了通过策略梯度方法设计具有任意外部与内部结构的自由形态机器人。该方法通过执行沉积或移除原子级构建模块束的动作，形成更高层次的非参数化宏观结构（如附肢、器官及空腔）。尽管本文仅提供开环控制的结果，但讨论了该方法未来如何适配闭环控制及实现从仿真到物理机器的sim2real迁移。