While neural implicit representations have gained popularity in multi-view 3D reconstruction, previous work struggles to yield physically plausible results, thereby limiting their applications in physics-demanding domains like embodied AI and robotics. The lack of plausibility originates from both the absence of physics modeling in the existing pipeline and their inability to recover intricate geometrical structures. In this paper, we introduce PhyRecon, which stands as the first approach to harness both differentiable rendering and differentiable physics simulation to learn implicit surface representations. Our framework proposes a novel differentiable particle-based physical simulator seamlessly integrated with the neural implicit representation. At its core is an efficient transformation between SDF-based implicit representation and explicit surface points by our proposed algorithm, Surface Points Marching Cubes (SP-MC), enabling differentiable learning with both rendering and physical losses. Moreover, we model both rendering and physical uncertainty to identify and compensate for the inconsistent and inaccurate monocular geometric priors. The physical uncertainty additionally enables a physics-guided pixel sampling to enhance the learning of slender structures. By amalgamating these techniques, our model facilitates efficient joint modeling with appearance, geometry, and physics. Extensive experiments demonstrate that PhyRecon significantly outperforms all state-of-the-art methods in terms of reconstruction quality. Our reconstruction results also yield superior physical stability, verified by Isaac Gym, with at least a 40% improvement across all datasets, opening broader avenues for future physics-based applications.

翻译：尽管神经隐式表示在多视图三维重建中已获得广泛应用，但现有方法难以生成物理合理的结果，从而限制了其在具身人工智能和机器人等对物理特性要求较高的领域中的应用。这种物理合理性的缺失源于现有流程中缺乏物理建模，以及无法恢复精细的几何结构。本文提出PhyRecon，这是首个利用可微渲染与可微物理仿真联合学习隐式表面表示的方法。我们的框架创新性地提出了一种可微的基于粒子的物理模拟器，并与神经隐式表示无缝集成。其核心是通过我们提出的表面点行进立方体（SP-MC）算法实现基于符号距离函数的隐式表示与显式表面点之间的高效转换，从而支持包含渲染损失和物理损失的可微学习。此外，我们对渲染不确定性和物理不确定性进行建模，以识别并补偿不一致且不准确单目几何先验。物理不确定性还实现了基于物理引导的像素采样，增强对细长结构的学习。通过融合这些技术，我们的模型实现了外观、几何与物理的高效联合建模。大量实验表明，PhyRecon在重建质量上显著优于所有现有最优方法。经Isaac Gym验证，我们的重建结果具有更优的物理稳定性，在所有数据集上至少提升40%，为未来基于物理的应用开辟了更广阔的路径。