We study the potential of noisy labels y to pretrain semantic segmentation models in a multi-modal learning framework for geospatial applications. Specifically, we propose a novel Cross-modal Sample Selection method (CromSS) that utilizes the class distributions P^{(d)}(x,c) over pixels x and classes c modelled by multiple sensors/modalities d of a given geospatial scene. Consistency of predictions across sensors $d$ is jointly informed by the entropy of P^{(d)}(x,c). Noisy label sampling we determine by the confidence of each sensor d in the noisy class label, P^{(d)}(x,c=y(x)). To verify the performance of our approach, we conduct experiments with Sentinel-1 (radar) and Sentinel-2 (optical) satellite imagery from the globally-sampled SSL4EO-S12 dataset. We pair those scenes with 9-class noisy labels sourced from the Google Dynamic World project for pretraining. Transfer learning evaluations (downstream task) on the DFC2020 dataset confirm the effectiveness of the proposed method for remote sensing image segmentation.

翻译：我们研究了在面向地理空间应用的多模态学习框架中，利用噪声标签 y 预训练语义分割模型的潜力。具体而言，我们提出了一种新颖的跨模态样本选择方法（CromSS），该方法利用给定地理场景中由多个传感器/模态 d 建模的像素 x 和类别 c 上的类别分布 P^{(d)}(x,c)。跨传感器 d 的预测一致性由 P^{(d)}(x,c) 的熵共同约束。噪声标签采样由每个传感器 d 对噪声类别标签的置信度 P^{(d)}(x,c=y(x)) 决定。为验证所提方法的性能，我们使用全球采样SSL4EO-S12数据集中的Sentinel-1（雷达）和Sentinel-2（光学）卫星影像进行实验，并将这些场景与来自Google Dynamic World项目的9类别噪声标签配对用于预训练。在DFC2020数据集上的迁移学习评估（下游任务）证实了该方法在遥感图像分割中的有效性。