When deploying segmentation models in practice, it is critical to evaluate their behaviors in varied and complex scenes. Different from the previous evaluation paradigms only in consideration of global attribute variations (e.g. adverse weather), we investigate both local and global attribute variations for robustness evaluation. To achieve this, we construct a mask-preserved attribute editing pipeline to edit visual attributes of real images with precise control of structural information. Therefore, the original segmentation labels can be reused for the edited images. Using our pipeline, we construct a benchmark covering both object and image attributes (e.g. color, material, pattern, style). We evaluate a broad variety of semantic segmentation models, spanning from conventional close-set models to recent open-vocabulary large models on their robustness to different types of variations. We find that both local and global attribute variations affect segmentation performances, and the sensitivity of models diverges across different variation types. We argue that local attributes have the same importance as global attributes, and should be considered in the robustness evaluation of segmentation models. Code: https://github.com/PRIS-CV/Pascal-EA.

翻译：在实际部署分割模型时，评估其在复杂多变场景中的表现至关重要。不同于以往仅考虑全局属性变化（例如恶劣天气）的评估范式，本研究同时探究局部与全局属性变化对模型鲁棒性的影响。为此，我们构建了一种基于掩码保留的属性编辑流程，可在精确控制结构信息的前提下编辑真实图像的视觉属性，从而使原始分割标签能够直接复用于编辑后的图像。通过该流程，我们建立了涵盖物体属性和图像属性（如颜色、材质、图案、风格）的基准测试集，并系统评估了从传统闭集模型到最新开放词汇大模型等多种语义分割模型在不同类型属性变化下的鲁棒性。研究发现，局部与全局属性变化均会影响分割性能，且模型对不同变化类型的敏感程度存在显著差异。我们认为局部属性与全局属性同等重要，应纳入分割模型鲁棒性评估体系。代码：https://github.com/PRIS-CV/Pascal-EA