Semantic segmentation methods have advanced significantly. Still, their robustness to real-world perturbations and object types not seen during training remains a challenge, particularly in safety-critical applications. We propose a novel approach to improve the robustness of semantic segmentation techniques by leveraging the synergy between label-to-image generators and image-to-label segmentation models. Specifically, we design Robusta, a novel robust conditional generative adversarial network to generate realistic and plausible perturbed images that can be used to train reliable segmentation models. We conduct in-depth studies of the proposed generative model, assess the performance and robustness of the downstream segmentation network, and demonstrate that our approach can significantly enhance the robustness in the face of real-world perturbations, distribution shifts, and out-of-distribution samples. Our results suggest that this approach could be valuable in safety-critical applications, where the reliability of perception modules such as semantic segmentation is of utmost importance and comes with a limited computational budget in inference. We release our code at https://github.com/ENSTA-U2IS/robusta.

翻译：语义分割方法已取得显著进展。然而，它们对现实世界扰动以及训练中未见物体类型的鲁棒性仍是一个挑战，尤其是在安全关键应用中。我们提出了一种新颖方法，通过利用标签到图像生成器与图像到标签分割模型之间的协同作用，提升语义分割技术的鲁棒性。具体而言，我们设计了Robusta——一种新颖的鲁棒条件生成对抗网络，用于生成可训练可靠分割模型的现实合理且带有扰动的图像。我们对所提出的生成模型进行了深入研究，评估了下游分割网络的性能与鲁棒性，并证明我们的方法能显著增强面对现实扰动、分布偏移及分布外样本时的鲁棒性。结果表明，该方法在安全关键应用中具有重要价值，此类应用中语义分割等感知模块的可靠性至关重要，且推理时计算预算有限。我们已在https://github.com/ENSTA-U2IS/robusta 发布代码。