3D scene understanding plays a vital role in vision-based autonomous driving. While most existing methods focus on 3D object detection, they have difficulty describing real-world objects of arbitrary shapes and infinite classes. Towards a more comprehensive perception of a 3D scene, in this paper, we propose a SurroundOcc method to predict the 3D occupancy with multi-camera images. We first extract multi-scale features for each image and adopt spatial 2D-3D attention to lift them to the 3D volume space. Then we apply 3D convolutions to progressively upsample the volume features and impose supervision on multiple levels. To obtain dense occupancy prediction, we design a pipeline to generate dense occupancy ground truth without expansive occupancy annotations. Specifically, we fuse multi-frame LiDAR scans of dynamic objects and static scenes separately. Then we adopt Poisson Reconstruction to fill the holes and voxelize the mesh to get dense occupancy labels. Extensive experiments on nuScenes and SemanticKITTI datasets demonstrate the superiority of our method. Code and dataset are available at https://github.com/weiyithu/SurroundOcc

翻译：3D场景理解在基于视觉的自动驾驶中扮演着关键角色。尽管现有方法大多聚焦于3D目标检测，但它们难以描述任意形状和无限类别的真实世界物体。为实现更全面的3D场景感知，本文提出SurroundOcc方法，通过多相机图像预测3D占据信息。我们首先为每张图像提取多尺度特征，并采用空间2D-3D注意力机制将其提升至3D体积空间。随后应用3D卷积逐步上采样体积特征，并在多个层级施加监督。为获取密集占据预测，我们设计了一套流程，无需昂贵的占据标注即可生成密集占据真值：具体而言，分别融合动态物体与静态场景的多帧激光雷达扫描数据，采用泊松重建填充空洞，并将网格体素化以生成密集占据标签。在nuScenes和SemanticKITTI数据集上的大量实验证明了我们方法的优越性。代码与数据集已开源至https://github.com/weiyithu/SurroundOcc。