Physics-based understanding of object interactions from sensory observations is an essential capability in augmented reality and robotics. It enables to capture the properties of a scene for simulation and control. In this paper, we propose a novel approach for real-to-sim which tracks rigid objects in 3D from RGB-D images and infers physical properties of the objects. We use a differentiable physics simulation as state-transition model in an Extended Kalman Filter which can model contact and friction for arbitrary mesh-based shapes and in this way estimate physically plausible trajectories. We demonstrate that our approach can filter position, orientation, velocities, and concurrently can estimate the coefficient of friction of the objects. We analyze our approach on various sliding scenarios in synthetic image sequences of single objects and colliding objects. We also demonstrate and evaluate our approach on a real-world dataset. We make our novel benchmark datasets publicly available to foster future research in this novel problem setting and comparison with our method.

翻译：从感官观测中实现基于物理的物体交互理解是增强现实与机器人领域的关键能力。该能力能够捕捉场景属性以支持仿真与控制。本文提出一种新颖的真实到仿真方法，该方法通过RGB-D图像在三维空间中追踪刚体目标，并推断物体的物理属性。我们在扩展卡尔曼滤波器中采用可微分物理仿真作为状态转移模型，该模型能够为任意基于网格的形状建模接触与摩擦效应，从而估计物理合理的运动轨迹。实验证明，我们的方法能够同步滤波目标的位置、姿态、速度，并同时估计物体的摩擦系数。我们在单物体滑动与碰撞物体的合成图像序列中对多种滑动场景进行了方法分析。此外，我们在真实世界数据集上验证并评估了所提方法。我们公开了全新的基准数据集，以促进这一新兴问题设置的后续研究，并为与本文方法的比较提供基准。