Open-vocabulary panoptic reconstruction offers comprehensive scene understanding, enabling advances in embodied robotics and photorealistic simulation. In this paper, we propose PanopticRecon++, an end-to-end method that formulates panoptic reconstruction through a novel cross-attention perspective. This perspective models the relationship between 3D instances (as queries) and the scene's 3D embedding field (as keys) through their attention map. Unlike existing methods that separate the optimization of queries and keys or overlook spatial proximity, PanopticRecon++ introduces learnable 3D Gaussians as instance queries. This formulation injects 3D spatial priors to preserve proximity while maintaining end-to-end optimizability. Moreover, this query formulation facilitates the alignment of 2D open-vocabulary instance IDs across frames by leveraging optimal linear assignment with instance masks rendered from the queries. Additionally, we ensure semantic-instance segmentation consistency by fusing query-based instance segmentation probabilities with semantic probabilities in a novel panoptic head supervised by a panoptic loss. During training, the number of instance query tokens dynamically adapts to match the number of objects. PanopticRecon++ shows competitive performance in terms of 3D and 2D segmentation and reconstruction performance on both simulation and real-world datasets, and demonstrates a user case as a robot simulator. Our project website is at: https://yuxuan1206.github.io/panopticrecon_pp/

翻译：开放词汇全景重建提供了全面的场景理解能力，推动了具身机器人学和逼真仿真技术的发展。本文提出PanopticRecon++，一种通过新颖的交叉注意力视角构建全景重建的端到端方法。该视角通过注意力图建模三维实例（作为查询）与场景三维嵌入场（作为键）之间的关系。与现有方法分离优化查询和键或忽略空间邻近性不同，PanopticRecon++引入可学习的三维高斯分布作为实例查询。这种构建方式注入了三维空间先验以保持邻近性，同时维持端到端的可优化性。此外，该查询构建通过利用最优线性分配与从查询渲染的实例掩码，促进了跨帧二维开放词汇实例ID的对齐。我们还通过在全景损失监督的新型全景头中融合基于查询的实例分割概率与语义概率，确保语义-实例分割的一致性。训练过程中，实例查询标记的数量动态适配以匹配物体数量。PanopticRecon++在仿真和真实数据集上均展现出具有竞争力的三维与二维分割及重建性能，并展示了作为机器人仿真器的应用案例。项目网站位于：https://yuxuan1206.github.io/panopticrecon_pp/