The mainstream CNN-based remote sensing (RS) image semantic segmentation approaches typically rely on massive labeled training data. Such a paradigm struggles with the problem of RS multi-view scene segmentation with limited labeled views due to the lack of considering 3D information within the scene. In this paper, we propose ''Implicit Ray-Transformer (IRT)'' based on Implicit Neural Representation (INR), for RS scene semantic segmentation with sparse labels (such as 4-6 labels per 100 images). We explore a new way of introducing multi-view 3D structure priors to the task for accurate and view-consistent semantic segmentation. The proposed method includes a two-stage learning process. In the first stage, we optimize a neural field to encode the color and 3D structure of the remote sensing scene based on multi-view images. In the second stage, we design a Ray Transformer to leverage the relations between the neural field 3D features and 2D texture features for learning better semantic representations. Different from previous methods that only consider 3D prior or 2D features, we incorporate additional 2D texture information and 3D prior by broadcasting CNN features to different point features along the sampled ray. To verify the effectiveness of the proposed method, we construct a challenging dataset containing six synthetic sub-datasets collected from the Carla platform and three real sub-datasets from Google Maps. Experiments show that the proposed method outperforms the CNN-based methods and the state-of-the-art INR-based segmentation methods in quantitative and qualitative metrics.

翻译：基于CNN的主流遥感图像语义分割方法通常依赖大量标注训练数据。由于未考虑场景中的三维信息，此类范式在标注视角有限的遥感多视角场景分割中面临挑战。本文提出基于隐式神经表示的"隐式射线变换器"，用于稀疏标注（如每100张图像仅含4-6个标注）的遥感场景语义分割。我们探索了一种将多视角三维结构先验引入该任务的新方法，以实现准确且视角一致的语义分割。所提方法包含两阶段学习过程：第一阶段，基于多视角图像优化神经场以编码遥感场景的颜色与三维结构；第二阶段，设计射线变换器利用神经场三维特征与二维纹理特征之间的关联学习更优的语义表示。与仅考虑三维先验或二维特征的先前方法不同，我们通过沿采样射线将CNN特征广播至不同点特征，融合了额外的二维纹理信息与三维先验。为验证方法的有效性，我们构建了一个包含Carla平台六个合成子数据集与Google Maps三个真实子数据集的挑战性数据集。实验表明，所提方法在定量与定性指标上均优于基于CNN的方法及当前最优的基于INR的分割方法。