The introduction of large, foundational models to computer vision has led to drastically improved performance on the task of semantic segmentation. However, these existing methods exhibit a large performance drop when testing on images degraded by weather conditions such as rain, fog, or snow. We introduce a general paired-training method that can be applied to all current foundational model architectures that leads to improved performance on images in adverse weather conditions. To this end, we create the WeatherProof Dataset, the first semantic segmentation dataset with accurate clear and adverse weather image pairs, which not only enables our new training paradigm, but also improves the evaluation of the performance gap between clear and degraded segmentation. We find that training on these paired clear and adverse weather frames which share an underlying scene results in improved performance on adverse weather data. With this knowledge, we propose a training pipeline which accentuates the advantages of paired-data training using consistency losses and language guidance, which leads to performance improvements by up to 18.4% as compared to standard training procedures.

翻译：大型基础模型引入计算机视觉领域大幅提升了语义分割任务的性能。然而，现有方法在测试受雨、雾、雪等天气条件退化的图像时表现出显著的性能下降。我们提出一种通用的配对训练方法，可应用于当前所有基础模型架构，从而改善恶劣天气条件下图像的处理性能。为此，我们构建了WeatherProof数据集——首个包含清晰与恶劣天气精准配对图像的语义分割数据集，该数据集不仅支持我们的新训练范式，还改进了对清晰与退化分割之间性能差距的评估。我们发现，在共享底层场景的清晰与恶劣天气配对帧上进行训练，能够提升恶劣天气数据的性能。基于此，我们提出一种训练流程，通过一致性损失和语言引导强化配对数据训练的优势，与标准训练流程相比，性能提升最高可达18.4%。