The transformation of features from 2D perspective space to 3D space is essential to multi-view 3D object detection. Recent approaches mainly focus on the design of view transformation, either pixel-wisely lifting perspective view features into 3D space with estimated depth or grid-wisely constructing BEV features via 3D projection, treating all pixels or grids equally. However, choosing what to transform is also important but has rarely been discussed before. The pixels of a moving car are more informative than the pixels of the sky. To fully utilize the information contained in images, the view transformation should be able to adapt to different image regions according to their contents. In this paper, we propose a novel framework named FrustumFormer, which pays more attention to the features in instance regions via adaptive instance-aware resampling. Specifically, the model obtains instance frustums on the bird's eye view by leveraging image view object proposals. An adaptive occupancy mask within the instance frustum is learned to refine the instance location. Moreover, the temporal frustum intersection could further reduce the localization uncertainty of objects. Comprehensive experiments on the nuScenes dataset demonstrate the effectiveness of FrustumFormer, and we achieve a new state-of-the-art performance on the benchmark. Codes and models will be made available at https://github.com/Robertwyq/Frustum.

翻译：从二维透视空间到三维空间的特征变换对于多视角三维目标检测至关重要。现有方法主要关注视角变换的设计，或通过估计深度将透视视图特征逐像素提升至三维空间，或通过三维投影以网格方式构建鸟瞰图特征，均将全部像素或网格同等对待。然而，选择需要变换的内容同样重要，却鲜有讨论。运动车辆像素所蕴含的信息远多于天空像素。为充分利用图像中的信息，视角变换需能根据图像区域内容自适应调整。本文提出名为FrustumFormer的新型框架，通过自适应实例感知重采样，更关注实例区域的特征。具体而言，模型利用图像视图的目标提议获取鸟瞰图上的实例平截锥体，并学习实例平截锥体内的自适应占用掩模以精炼实例位置。此外，时序平截锥体交会可进一步降低目标定位的不确定性。在nuScenes数据集上的综合实验证明了FrustumFormer的有效性，我们在该基准上取得了最新的最佳性能。代码与模型将开源至https://github.com/Robertwyq/Frustum。