For robots to robustly understand and interact with the physical world, it is highly beneficial to have a comprehensive representation - modelling geometry, physics, and visual observations - that informs perception, planning, and control algorithms. We propose a novel dual Gaussian-Particle representation that models the physical world while (i) enabling predictive simulation of future states and (ii) allowing online correction from visual observations in a dynamic world. Our representation comprises particles that capture the geometrical aspect of objects in the world and can be used alongside a particle-based physics system to anticipate physically plausible future states. Attached to these particles are 3D Gaussians that render images from any viewpoint through a splatting process thus capturing the visual state. By comparing the predicted and observed images, our approach generates visual forces that correct the particle positions while respecting known physical constraints. By integrating predictive physical modelling with continuous visually-derived corrections, our unified representation reasons about the present and future while synchronizing with reality. Our system runs in realtime at 30Hz using only 3 cameras. We validate our approach on 2D and 3D tracking tasks as well as photometric reconstruction quality. Videos are found at https://embodied-gaussians.github.io/.

翻译：为使机器人能够稳健地理解物理世界并与之交互，构建一个融合几何、物理及视觉观测的综合表征体系至关重要，该体系应为感知、规划与控制算法提供信息支撑。本文提出一种新颖的双重高斯-粒子表征方法，该模型在刻画物理世界的同时具备以下特性：(i) 能够对未来状态进行预测性仿真；(ii) 支持在动态环境中基于视觉观测进行在线校正。我们的表征体系由捕捉物体几何特性的粒子构成，这些粒子可与基于粒子的物理系统协同工作，以预测符合物理规律的未来状态。附着于这些粒子的三维高斯模型通过溅射过程可从任意视角渲染图像，从而捕捉视觉状态。通过比较预测图像与观测图像，我们的方法生成视觉作用力，在遵循已知物理约束的条件下校正粒子位置。通过将预测性物理建模与持续视觉校正相结合，我们的统一表征体系能够同步现实状态并对当前及未来状态进行推演。该系统仅需3个摄像头即可实现30Hz的实时运行。我们在二维与三维跟踪任务以及光度重建质量评估中验证了所提方法的有效性。演示视频详见 https://embodied-gaussians.github.io/。