Assessing the reliability of perception models to covariate shifts and out-of-distribution (OOD) detection is crucial for safety-critical applications such as autonomous vehicles. By nature of the task, however, the relevant data is difficult to collect and annotate. In this paper, we challenge cutting-edge generative models to automatically synthesize data for assessing reliability in semantic segmentation. By fine-tuning Stable Diffusion, we perform zero-shot generation of synthetic data in OOD domains or inpainted with OOD objects. Synthetic data is employed to provide an initial assessment of pretrained segmenters, thereby offering insights into their performance when confronted with real edge cases. Through extensive experiments, we demonstrate a high correlation between the performance on synthetic data and the performance on real OOD data, showing the validity approach. Furthermore, we illustrate how synthetic data can be utilized to enhance the calibration and OOD detection capabilities of segmenters.

翻译：评估感知模型在协变量偏移和分布外（OOD）检测中的可靠性，对自动驾驶等安全关键型应用至关重要。然而，由于任务本身的特性，相关数据的采集和标注十分困难。本文挑战前沿生成模型，使其自动合成数据，以评估语义分割的可靠性。通过微调Stable Diffusion，我们在OOD域内实现合成数据的零样本生成，或通过修复方式添加OOD对象。合成数据可用于对预训练分割器进行初步评估，从而洞察其在面对真实边缘案例时的表现。通过大量实验，我们证明合成数据上的性能与真实OOD数据上的性能之间存在高度相关性，验证了该方法的有效性。此外，我们展示了如何利用合成数据增强分割器的校准能力与OOD检测能力。