This paper raises the new task of Fisheye Semantic Completion (FSC), where dense texture, structure, and semantics of a fisheye image are inferred even beyond the sensor field-of-view (FoV). Fisheye cameras have larger FoV than ordinary pinhole cameras, yet its unique special imaging model naturally leads to a blind area at the edge of the image plane. This is suboptimal for safety-critical applications since important perception tasks, such as semantic segmentation, become very challenging within the blind zone. Previous works considered the out-FoV outpainting and in-FoV segmentation separately. However, we observe that these two tasks are actually closely coupled. To jointly estimate the tightly intertwined complete fisheye image and scene semantics, we introduce the new FishDreamer which relies on successful ViTs enhanced with a novel Polar-aware Cross Attention module (PCA) to leverage dense context and guide semantically-consistent content generation while considering different polar distributions. In addition to the contribution of the novel task and architecture, we also derive Cityscapes-BF and KITTI360-BF datasets to facilitate training and evaluation of this new track. Our experiments demonstrate that the proposed FishDreamer outperforms methods solving each task in isolation and surpasses alternative approaches on the Fisheye Semantic Completion. Code and datasets are publicly available at https://github.com/MasterHow/FishDreamer.

翻译：本文提出鱼眼语义补全（Fisheye Semantic Completion, FSC）这一新任务，旨在推断鱼眼图像中超出传感器视场角（field-of-view, FoV）的密集纹理、结构与语义信息。鱼眼相机相比普通针孔相机具有更广的视场角，但其特殊的成像模型自然导致图像平面边缘存在盲区。这对安全关键型应用而言并不理想，因为在盲区内诸如语义分割等重要感知任务变得极具挑战性。以往研究将视场外外推与视场内分割作为独立任务处理。然而，我们观察到这两个任务实际上紧密耦合。为联合估计相互交织的完整鱼眼图像与场景语义，我们提出新的鱼眼梦想家（FishDreamer）模型，其基于成功的视觉Transformer架构，并融入新颖的极坐标感知交叉注意力模块（Polar-aware Cross Attention module, PCA），以利用密集上下文信息并在考虑不同极坐标分布的同时引导语义一致的内容生成。除了新任务与架构的贡献外，我们还构建了Cityscapes-BF和KITTI360-BF数据集，以促进该新方向的训练与评估。实验表明，所提出的鱼眼梦想家优于分别解决各独立任务的方法，并在鱼眼语义补全任务上超越替代方案。代码与数据集已公开发布于https://github.com/MasterHow/FishDreamer。