We present a differentiable pipeline for simulating the motion of objects that represent their geometry as a continuous density field parameterized as a deep network. This includes Neural Radiance Fields (NeRFs), and other related models. From the density field, we estimate the dynamical properties of the object, including its mass, center of mass, and inertia matrix. We then introduce a differentiable contact model based on the density field for computing normal and friction forces resulting from collisions. This allows a robot to autonomously build object models that are visually and \emph{dynamically} accurate from still images and videos of objects in motion. The resulting Dynamics-Augmented Neural Objects (DANOs) are simulated with an existing differentiable simulation engine, Dojo, interacting with other standard simulation objects, such as spheres, planes, and robots specified as URDFs. A robot can use this simulation to optimize grasps and manipulation trajectories of neural objects, or to improve the neural object models through gradient-based real-to-simulation transfer. We demonstrate the pipeline to learn the coefficient of friction of a bar of soap from a real video of the soap sliding on a table. We also learn the coefficient of friction and mass of a Stanford bunny through interactions with a Panda robot arm from synthetic data, and we optimize trajectories in simulation for the Panda arm to push the bunny to a goal location.

翻译：我们提出了一种可微分的流水线，用于模拟以连续密度场表示几何形状的对象运动，该密度场通过深度网络参数化，包括神经辐射场（NeRF）及其他相关模型。基于密度场，我们估算对象的动力学属性，包括质量、质心和惯性矩阵。随后，我们引入一种基于密度场的可微分接触模型，用于计算碰撞产生的法向力和摩擦力。这使得机器人能够通过静止图像和运动中的对象视频自主构建视觉和动力学精确的对象模型。所得增强动力学的神经对象（DANOs）可利用现有可微分仿真引擎Dojo进行模拟，并与球体、平面及以URDF格式指定的机器人等其他标准仿真对象交互。机器人可利用此仿真优化神经对象的抓取和操作轨迹，或通过基于梯度的实况到仿真转移改进神经对象模型。我们演示了该流水线：通过一段肥皂在桌面上滑动的真实视频学习其摩擦系数；同时，利用合成数据通过Panda机器人手臂与斯坦福兔子的交互学习其摩擦系数和质量；并在仿真中优化Panda手臂将兔子推至目标位置的轨迹。