We introduce Structure from Action (SfA), a framework to discover 3D part geometry and joint parameters of unseen articulated objects via a sequence of inferred interactions. Our key insight is that 3D interaction and perception should be considered in conjunction to construct 3D articulated CAD models, especially for categories not seen during training. By selecting informative interactions, SfA discovers parts and reveals occluded surfaces, like the inside of a closed drawer. By aggregating visual observations in 3D, SfA accurately segments multiple parts, reconstructs part geometry, and infers all joint parameters in a canonical coordinate frame. Our experiments demonstrate that a SfA model trained in simulation can generalize to many unseen object categories with diverse structures and to real-world objects. Empirically, SfA outperforms a pipeline of state-of-the-art components by 25.4 3D IoU percentage points on unseen categories, while matching already performant joint estimation baselines.

翻译：我们提出“从行动中理解结构”（Structure from Action, SfA）框架，该框架通过一系列推断的交互行为，发现未知铰接物体的三维部件几何结构与关节参数。核心洞见在于：构建三维铰接CAD模型时，特别是针对训练阶段未见过的物体类别，需将三维交互与感知视为联合整体。通过选择信息量丰富的交互动作，SfA能逐步发现部件并揭示被遮挡表面（如关闭抽屉的内部结构）。通过聚合三维空间中的视觉观测，SfA可精确分割多个部件、重建部件几何形状，并推断规范坐标框架下的全部关节参数。实验表明，在仿真环境中训练的SfA模型可泛化至结构多样的未知物体类别与真实世界物体。实证结果显示，在未知类别物体上，SfA相较由顶级组件构成的基线系统，三维交并比（3D IoU）提升25.4个百分点，同时达到与现有高性能关节估计方法相当的水平。