Physics-based understanding of object interactions from sensory observations is an essential capability in augmented reality and robotics. It enables capturing the properties of a scene for simulation and control. In this paper, we propose a novel approach for real-to-sim which tracks rigid objects in 3D from RGB-D images and infers physical properties of the objects. We use a differentiable physics simulation as state-transition model in an Extended Kalman Filter which can model contact and friction for arbitrary mesh-based shapes and in this way estimate physically plausible trajectories. We demonstrate that our approach can filter position, orientation, velocities, and concurrently can estimate the coefficient of friction of the objects. We analyse our approach on various sliding scenarios in synthetic image sequences of single objects and colliding objects. We also demonstrate and evaluate our approach on a real-world dataset. We will make our novel benchmark datasets publicly available to foster future research in this novel problem setting and comparison with our method.

翻译：物理感知的对象交互理解是增强现实和机器人技术中的关键能力，它能够捕捉场景属性以用于仿真和控制。本文提出了一种新颖的"实到仿"方法，通过RGB-D图像对三维空间中的刚体目标进行跟踪，并推断其物理属性。我们采用可微物理仿真作为扩展卡尔曼滤波中的状态转移模型，该模型能够处理任意网格形状的接触与摩擦，从而估计符合物理规律的轨迹。实验证明，该方法不仅能过滤位置、姿态和速度信息，还能同步估计目标的摩擦系数。我们在包含单目标和碰撞目标的合成图像序列中，针对多种滑动场景进行了分析，并在真实数据集上验证了方法的有效性。为促进该新问题领域的后续研究及方法比较，我们将公开所构建的新型基准数据集。