When deploying a semantic segmentation model into the real world, it will inevitably be confronted with semantic classes unseen during training. Thus, to safely deploy such systems, it is crucial to accurately evaluate and improve their anomaly segmentation capabilities. However, acquiring and labelling semantic segmentation data is expensive and unanticipated conditions are long-tail and potentially hazardous. Indeed, existing anomaly segmentation datasets capture a limited number of anomalies, lack realism or have strong domain shifts. In this paper, we propose the Placing Objects in Context (POC) pipeline to realistically add any object into any image via diffusion models. POC can be used to easily extend any dataset with an arbitrary number of objects. In our experiments, we present different anomaly segmentation datasets based on POC-generated data and show that POC can improve the performance of recent state-of-the-art anomaly fine-tuning methods in several standardized benchmarks. POC is also effective to learn new classes. For example, we use it to edit Cityscapes samples by adding a subset of Pascal classes and show that models trained on such data achieve comparable performance to the Pascal-trained baseline. This corroborates the low sim-to-real gap of models trained on POC-generated images.

翻译：在现实世界中部署语义分割模型时，模型将不可避免地遇到训练过程中未见过的语义类别。因此，为确保此类系统的安全部署，准确评估并改进其异常分割能力至关重要。然而，获取和标注语义分割数据成本高昂，且不可预见的情况往往具有长尾分布和潜在危害性。事实上，现有的异常分割数据集捕获的异常数量有限，缺乏真实性，或存在强烈的域偏移。本文提出了一种“在上下文中放置物体”（POC）流程，该流程利用扩散模型将任意物体真实地添加到任意图像中。POC可用于轻松扩展任何数据集，添加任意数量的物体。在实验中，我们展示了基于POC生成数据的不同异常分割数据集，并证明POC能够在多个标准化基准测试中提升最新异常微调方法的性能。此外，POC还可用于学习新类别。例如，我们通过添加部分Pascal类别来编辑Cityscapes样本，实验表明，基于此类数据训练的模型达到了与Pascal训练基线相当的性能。这证实了基于POC生成图像训练的模型具有较低的模拟到真实差距。