Physical simulations that accurately model reality are crucial for many engineering disciplines such as mechanical engineering and robotic motion planning. In recent years, learned Graph Network Simulators produced accurate mesh-based simulations while requiring only a fraction of the computational cost of traditional simulators. Yet, the resulting predictors are confined to learning from data generated by existing mesh-based simulators and thus cannot include real world sensory information such as point cloud data. As these predictors have to simulate complex physical systems from only an initial state, they exhibit a high error accumulation for long-term predictions. In this work, we integrate sensory information to ground Graph Network Simulators on real world observations. In particular, we predict the mesh state of deformable objects by utilizing point cloud data. The resulting model allows for accurate predictions over longer time horizons, even under uncertainties in the simulation, such as unknown material properties. Since point clouds are usually not available for every time step, especially in online settings, we employ an imputation-based model. The model can make use of such additional information only when provided, and resorts to a standard Graph Network Simulator, otherwise. We experimentally validate our approach on a suite of prediction tasks for mesh-based interactions between soft and rigid bodies. Our method results in utilization of additional point cloud information to accurately predict stable simulations where existing Graph Network Simulators fail.

翻译：能够精确模拟现实世界的物理模拟对于机械工程和机器人运动规划等许多工程学科至关重要。近年来，学习型图网络模拟器实现了精确的基于网格的模拟，同时仅需传统模拟器计算成本的一小部分。然而，由此产生的预测器仅限于从现有基于网格的模拟器生成的数据中学习，因此无法包含诸如点云数据等真实世界感官信息。由于这些预测器必须仅从初始状态模拟复杂物理系统，它们在长期预测中表现出较高的误差累积。在这项工作中，我们整合感官信息以将图网络模拟器接地到真实世界观测上。具体而言，我们利用点云数据预测可变形物体的网格状态。所得模型即使在模拟中存在不确定性（如未知材料属性）时，也能在更长的时间范围内实现精确预测。由于点云数据并非每个时间步都可用（尤其是在在线设置中），我们采用基于插补的模型。该模型仅在有额外信息时加以利用，否则退化为标准图网络模拟器。我们在一套用于软硬体之间基于网格交互的预测任务上实验验证了我们的方法。我们的方法能够利用额外的点云信息，精确预测现有图网络模拟器失败的稳定模拟。