Physical simulations that accurately model reality are crucial for many engineering disciplines such as mechanical engineering and robotic motion planning. In recent years, learned Graph Network Simulators produced accurate mesh-based simulations while requiring only a fraction of the computational cost of traditional simulators. Yet, the resulting predictors are confined to learning from data generated by existing mesh-based simulators and thus cannot include real world sensory information such as point cloud data. As these predictors have to simulate complex physical systems from only an initial state, they exhibit a high error accumulation for long-term predictions. In this work, we integrate sensory information to ground Graph Network Simulators on real world observations. In particular, we predict the mesh state of deformable objects by utilizing point cloud data. The resulting model allows for accurate predictions over longer time horizons, even under uncertainties in the simulation, such as unknown material properties. Since point clouds are usually not available for every time step, especially in online settings, we employ an imputation-based model. The model can make use of such additional information only when provided, and resorts to a standard Graph Network Simulator, otherwise. We experimentally validate our approach on a suite of prediction tasks for mesh-based interactions between soft and rigid bodies. Our method results in utilization of additional point cloud information to accurately predict stable simulations where existing Graph Network Simulators fail.

翻译：物理仿真对于机械工程和机器人运动规划等许多工程学科至关重要，它能准确模拟现实世界。近年来，学习型图网络仿真器能够生成精确的网格仿真，且计算成本仅为传统仿真器的一小部分。然而，由此产生的预测模型仅限于从现有网格仿真器生成的数据中学习，因此无法融入真实世界的感知信息，例如点云数据。由于这些预测模型只能从初始状态模拟复杂物理系统，它们在长期预测中表现出较高的误差积累。在本工作中，我们整合感知信息，以基于真实世界观测对标定图网络仿真器。具体而言，我们利用点云数据预测可变形物体的网格状态。即使在仿真存在不确定性（如未知材料属性）的情况下，所得模型也能实现更长时间范围内的精确预测。由于点云通常并非每个时间步都可用（尤其是在在线场景下），我们采用基于插补的模型。该模型仅在提供此类额外信息时加以利用，否则退化为标准图网络仿真器。我们在一系列基于网格的软体与刚体交互预测任务上实验验证了该方法。我们的方法利用额外的点云信息，能够准确预测现有图网络仿真器无法实现的稳定仿真结果。