The advent of high-resolution multispectral/hyperspectral sensors, LiDAR DSM (Digital Surface Model) information and many others has provided us with an unprecedented wealth of data for Earth Observation. Multimodal AI seeks to exploit those complementary data sources, particularly for complex tasks like semantic segmentation. While specialized architectures have been developed, they are highly complicated via significant effort in model design, and require considerable re-engineering whenever a new modality emerges. Recent trends in general-purpose multimodal networks have shown great potential to achieve state-of-the-art performance across multiple multimodal tasks with one unified architecture. In this work, we investigate the performance of PerceiverIO, one in the general-purpose multimodal family, in the remote sensing semantic segmentation domain. Our experiments reveal that this ostensibly universal network struggles with object scale variation in remote sensing images and fails to detect the presence of cars from a top-down view. To address these issues, even with extreme class imbalance issues, we propose a spatial and volumetric learning component. Specifically, we design a UNet-inspired module that employs 3D convolution to encode vital local information and learn cross-modal features simultaneously, while reducing network computational burden via the cross-attention mechanism of PerceiverIO. The effectiveness of the proposed component is validated through extensive experiments comparing it with other methods such as 2D convolution, and dual local module (\ie the combination of Conv2D 1x1 and Conv2D 3x3 inspired by UNetFormer). The proposed method achieves competitive results with specialized architectures like UNetFormer and SwinUNet, showing its potential to minimize network architecture engineering with a minimal compromise on the performance.

翻译：高分辨率多光谱/高光谱传感器、LiDAR DSM（数字表面模型）信息及其他多种数据的出现，为地球观测提供了前所未有的数据丰富性。多模态人工智能旨在利用这些互补数据源，特别是在语义分割等复杂任务中。尽管已有专门的架构被开发出来，但它们通过大量的模型设计工作变得极为复杂，并且每当出现新的模态时都需要进行大量的重新工程。近期通用多模态网络的趋势表明，通过单一统一架构在多个多模态任务上实现最先进性能的巨大潜力。在本工作中，我们研究了通用多模态家族中的PerceiverIO在遥感语义分割领域的性能。实验揭示，这一表面上通用的网络在处理遥感图像中的物体尺度变化时存在困难，并且无法从俯视视角检测到车辆的存在。为解决这些问题，即使面临极端类别不平衡，我们提出了一种空间与体积学习组件。具体而言，我们设计了一个受UNet启发的模块，该模块利用3D卷积同时编码关键的局部信息并学习跨模态特征，同时通过PerceiverIO的交叉注意力机制降低网络计算负担。通过与其他方法（如2D卷积和受UNetFormer启发的双局部模块（即Conv2D 1x1与Conv2D 3x3的组合））的广泛对比实验，验证了所提组件的有效性。所提方法在性能上达到了与UNetFormer和SwinUNet等专门架构相竞争的结果，展示了其以最小性能折衷实现最小化网络架构工程的潜力。