Estimating simulation-ready scenes from real-world observations is crucial for downstream planning and policy learning tasks. Regretfully, existing methods struggle in cluttered environments, often exhibiting prohibitive computational cost, poor robustness, and restricted generality when scaling to multiple interacting objects. We propose a unified optimization-based formulation for real-to-sim scene estimation that jointly recovers the shapes and poses of multiple rigid objects under physical constraints. Our method is built on two key technical innovations. First, we leverage the recently introduced shape-differentiable contact model, whose global differentiability permits joint optimization over object geometry and pose while modeling inter-object contacts. Second, we exploit the structured sparsity of the augmented Lagrangian Hessian to derive an efficient linear system solver whose computational cost scales favorably with scene complexity. Building on this formulation, we develop an end-to-end real-to-sim scene estimation pipeline that integrates learning-based object initialization, physics-constrained joint shape-pose optimization, and differentiable texture refinement. Experiments on cluttered scenes with up to 5 objects and 22 convex hulls demonstrate that our approach robustly reconstructs physically valid, simulation-ready object shapes and poses.

翻译：从真实世界观测中估计可直接用于仿真的场景，对于下游规划与策略学习任务至关重要。遗憾的是，现有方法在杂乱环境中表现不佳，当扩展到多个相互作用的物体时，常面临计算成本过高、鲁棒性差以及泛化能力受限等问题。本文提出一种基于优化的统一框架，用于实现从真实到仿真的场景估计，该框架能够在物理约束下联合恢复多个刚性物体的形状与位姿。我们的方法建立在两项关键技术创新的基础上。首先，我们利用了近期提出的形状可微分接触模型，其全局可微性允许我们在建模物体间接触的同时，对物体几何与位姿进行联合优化。其次，我们利用增广拉格朗日函数海森矩阵的结构化稀疏性，推导出一种高效的线性系统求解器，其计算成本随场景复杂度的增加而呈现有利的缩放特性。基于此框架，我们开发了一个端到端的从真实到仿真场景估计流程，该流程集成了基于学习的物体初始化、物理约束下的联合形状-位姿优化以及可微分纹理细化。在包含多达5个物体和22个凸包的杂乱场景上进行的实验表明，我们的方法能够鲁棒地重建出物理有效且可直接用于仿真的物体形状与位姿。