Semantic segmentation has recently witnessed great progress. Despite the impressive overall results, the segmentation performance in some hard areas (e.g., small objects or thin parts) is still not promising. A straightforward solution is hard sample mining, which is widely used in object detection. Yet, most existing hard pixel mining strategies for semantic segmentation often rely on pixel's loss value, which tends to decrease during training. Intuitively, the pixel hardness for segmentation mainly depends on image structure and is expected to be stable. In this paper, we propose to learn pixel hardness for semantic segmentation, leveraging hardness information contained in global and historical loss values. More precisely, we add a gradient-independent branch for learning a hardness level (HL) map by maximizing hardness-weighted segmentation loss, which is minimized for the segmentation head. This encourages large hardness values in difficult areas, leading to appropriate and stable HL map. Despite its simplicity, the proposed method can be applied to most segmentation methods with no and marginal extra cost during inference and training, respectively. Without bells and whistles, the proposed method achieves consistent/significant improvement (1.37% mIoU on average) over most popular semantic segmentation methods on Cityscapes dataset, and demonstrates good generalization ability across domains. The source codes are available at https://github.com/Menoly-xin/Hardness-Level-Learning .

翻译：语义分割近年来取得了显著进展。尽管总体结果令人印象深刻，但在一些困难区域（如小物体或薄部分）的分割性能仍不理想。一种直接的解决方案是困难样本挖掘，这在目标检测中广泛使用。然而，现有的大多数语义分割硬像素挖掘策略通常依赖于像素的损失值，而该值在训练过程中会趋于减小。直观上，分割中的像素硬度主要取决于图像结构，且应保持稳定。在本文中，我们提出学习语义分割中的像素硬度，利用全局和历史损失值中包含的硬度信息。更具体地说，我们添加了一个与梯度无关的分支，通过最大化硬度加权的分割损失来学习硬度级别（HL）图，而该损失对于分割头是最小化的。这促使困难区域具有较大的硬度值，从而生成适当且稳定的HL图。尽管方法简单，但所提出的方法可以应用于大多数分割方法，在推理和训练过程中分别无需或仅需少量额外成本。在不添加额外复杂组件的情况下，该方法在Cityscapes数据集上对大多数流行的语义分割方法实现了一致且显著的改进（平均mIoU提升1.37%），并展现出良好的跨域泛化能力。源代码可在https://github.com/Menoly-xin/Hardness-Level-Learning获取。