Neural rendering has made significant strides in 3D reconstruction and novel view synthesis. With the integration with physics, it opens up new applications. The inverse problem of estimating physics from visual data, however, still remains challenging, limiting its effectiveness for applications like physically accurate digital twin creation in robotics and XR. Existing methods that incorporate physics into neural rendering frameworks typically require dense multi-view videos as input, making them impractical for scalable, real-world use. When presented with sparse multi-view videos, the sequential optimization strategy used by existing approaches introduces significant error accumulation, e.g., poor initial 3D reconstruction leads to bad material parameter estimation in subsequent stages. Instead of sequential optimization, directly optimizing all parameters at the same time also fails due to the highly non-convex and often non-differentiable nature of the problem. We propose ProJo4D, a progressive joint optimization framework that gradually increases the set of jointly optimized parameters guided by their sensitivity, leading to fully joint optimization over geometry, appearance, physical state, and material property. Evaluations on PAC-NeRF and Spring-Gaus datasets show that ProJo4D outperforms prior work in 4D future state prediction, novel view rendering of future state, and material parameter estimation, demonstrating its effectiveness in physically grounded 4D scene understanding. For demos, please visit the project webpage: https://daniel03c1.github.io/ProJo4D/

翻译：神经渲染在三维重建和新视角合成领域已取得显著进展。通过与物理学的结合，它开启了新的应用领域。然而，从视觉数据中估计物理参数的逆问题仍然具有挑战性，限制了其在机器人学和扩展现实（XR）等领域创建物理精确数字孪生等应用中的有效性。现有将物理学融入神经渲染框架的方法通常需要密集的多视角视频作为输入，这使其难以扩展到实际应用中。当输入为稀疏多视角视频时，现有方法采用的顺序优化策略会引入显著的误差累积，例如，较差的三维重建初始结果会导致后续阶段的材料参数估计失败。与顺序优化不同，直接同时优化所有参数也会失败，因为该问题具有高度非凸性且通常不可微。我们提出了ProJo4D，一种渐进式联合优化框架，它根据参数的敏感度指导，逐步增加联合优化的参数集合，最终实现对几何、外观、物理状态和材料属性的完全联合优化。在PAC-NeRF和Spring-Gaus数据集上的评估表明，ProJo4D在四维未来状态预测、未来状态的新视角渲染以及材料参数估计方面均优于现有方法，证明了其在基于物理的四维场景理解方面的有效性。演示请访问项目网页：https://daniel03c1.github.io/ProJo4D/