Visual understanding and segmentation of materials and their states is fundamental to understanding the physical world. The myriad textures, shapes, and often blurry boundaries formed by materials make this task particularly hard to generalize. Whether it's identifying wet regions of a surface, minerals in rocks, infected regions in plants, or pollution in water, each material state has its own unique form. For neural nets to learn general class-agnostic material segmentation, it is necessary to first collect and annotate data that captures this complexity. Collecting and manually annotating real-world images is limited by the cost and precision of manual labor. In contrast, synthetic CGI data is highly accurate and almost cost-free, but fails to replicate the vast diversity of the material world. This work offers a method to bridge this crucial gap by implanting patterns extracted from real-world images in synthetic data. Hence, patterns automatically collected from natural images are used to map materials into synthetic scenes. This unsupervised approach allows the generated data to capture the vast complexity of the real world while maintaining the precision and scale of synthetic data. We also present the first general benchmark for zero-shot material state segmentation. The benchmark contains a wide range of real-world images of material states, like food, rocks, construction, plants, liquids, and many others, each in various states (wet/dry/stained/cooked/burned/worn/rusted/sediment/foam, etc.). The annotation includes both partial similarity between regions with similar but not identical materials, and hard segmentation of only points in the exact same material state. We show that net trains on MatSeg significantly outperform existing state-of-the-art methods on this task. The dataset, code, and trained model are available

翻译：对材料及其状态的视觉理解和分割是理解物理世界的基础。材料构成的各种纹理、形状及其常有的模糊边界使得这一任务特别难以泛化。无论是识别表面的湿润区域、岩石中的矿物、植物的感染区域，还是水中的污染物，每种材料状态都有其独特的表现形式。要让神经网络学习通用的、与类别无关的材料分割，首先需要收集并标注能够捕捉这种复杂性的数据。收集并手动标注真实世界图像受限于人工成本与精度。相比之下，合成CGI数据高度精确且几乎零成本，但无法复现材料世界的巨大多样性。本文提出了一种方法，通过将从真实世界图像中提取的模式植入合成数据，来弥合这一关键差距。因此，使用从自然图像中自动收集的模式将材料映射到合成场景中。这种无监督方法使生成的数据能够捕捉真实世界的巨大复杂性，同时保持合成数据的精度和规模。我们还首次提出了用于零样本材料状态分割的通用基准。该基准包含广泛的真实世界材料状态图像，如食物、岩石、建筑、植物、液体等，每种状态均呈现不同形态（湿/干/脏污/煮熟/烧焦/磨损/生锈/沉积/泡沫等）。标注内容包括区域间部分相似性（材料相似但非完全相同）以及仅对完全相同材料状态点的严格分割。我们证明，在该基准上训练的模型（MatSeg）显著优于现有最先进方法。数据集、代码及训练模型均已公开。