Semantic segmentation is a fundamental task in visual scene understanding. We focus on the supervised setting, where ground-truth semantic annotations are available. Based on knowledge about the high regularity of real-world scenes, we propose a method for improving class predictions by learning to selectively exploit information from neighboring pixels. In particular, our method is based on the prior that for each pixel, there is a seed pixel in its close neighborhood sharing the same prediction with the former. Motivated by this prior, we design a novel two-head network, named Offset Vector Network (OVeNet), which generates both standard semantic predictions and a dense 2D offset vector field indicating the offset from each pixel to the respective seed pixel, which is used to compute an alternative, seed-based semantic prediction. The two predictions are adaptively fused at each pixel using a learnt dense confidence map for the predicted offset vector field. We supervise offset vectors indirectly via optimizing the seed-based prediction and via a novel loss on the confidence map. Compared to the baseline state-of-the-art architectures HRNet and HRNet+OCR on which OVeNet is built, the latter achieves significant performance gains on two prominent benchmarks for semantic segmentation of driving scenes, namely Cityscapes and ACDC. Code is available at https://github.com/stamatisalex/OVeNet

翻译：语义分割是视觉场景理解中的基础任务。我们关注有监督设置，即真实语义标注可用的情况。基于现实世界场景高度规律性的知识，我们提出一种通过学习选择性利用邻域像素信息来改进类别预测的方法。具体而言，我们的方法基于以下先验：每个像素在其近邻中存在一个共享相同预测的种子像素。受此先验启发，我们设计了一种新颖的双头网络，命名为偏移向量网络（OVeNet），该网络同时生成标准语义预测和一个稠密的二维偏移向量场，该向量场表示每个像素到对应种子像素的偏移量，并用于计算基于种子的替代语义预测。利用学习到的预测偏移向量场的稠密置信图，这两种预测在每个像素处被自适应地融合。我们通过优化基于种子的预测以及针对置信图设计的新型损失函数来间接监督偏移向量。与构建OVeNet所基于的基线最先进架构HRNet和HRNet+OCR相比，OVeNet在驾驶场景语义分割的两个重要基准（即Cityscapes和ACDC）上取得了显著的性能提升。代码已开源：https://github.com/stamatisalex/OVeNet