Toward desirable saliency prediction, the types and numbers of inputs for a salient object detection (SOD) algorithm may dynamically change in many real-life applications. However, existing SOD algorithms are mainly designed or trained for one particular type of inputs, failing to be generalized to other types of inputs. Consequentially, more types of SOD algorithms need to be prepared in advance for handling different types of inputs, raising huge hardware and research costs. Differently, in this paper, we propose a new type of SOD task, termed Arbitrary Modality SOD (AM SOD). The most prominent characteristics of AM SOD are that the modality types and modality numbers will be arbitrary or dynamically changed. The former means that the inputs to the AM SOD algorithm may be arbitrary modalities such as RGB, depths, or even any combination of them. While, the latter indicates that the inputs may have arbitrary modality numbers as the input type is changed, e.g. single-modality RGB image, dual-modality RGB-Depth (RGB-D) images or triple-modality RGB-Depth-Thermal (RGB-D-T) images. Accordingly, a preliminary solution to the above challenges, \i.e. a modality switch network (MSN), is proposed in this paper. In particular, a modality switch feature extractor (MSFE) is first designed to extract discriminative features from each modality effectively by introducing some modality indicators, which will generate some weights for modality switching. Subsequently, a dynamic fusion module (DFM) is proposed to adaptively fuse features from a variable number of modalities based on a novel Transformer structure. Finally, a new dataset, named AM-XD, is constructed to facilitate research on AM SOD. Extensive experiments demonstrate that our AM SOD method can effectively cope with changes in the type and number of input modalities for robust salient object detection.

翻译：为达到理想的显著性预测效果，显著性目标检测（SOD）算法在实际应用中面临的输入类型与数量可能动态变化。然而现有SOD算法主要针对特定输入类型设计或训练，难以泛化至其他输入类型。这导致需预先准备多种SOD算法以处理不同输入类型，大幅增加硬件与研究成本。为此，本文提出一种新型SOD任务，即任意模态显著性目标检测（AM SOD）。AM SOD最显著的特征是模态类型与模态数量具有任意性或动态变化性：前者指算法输入可为任意模态（如RGB、深度图或任意组合），后者指输入模态数量会随类型变化而改变（例如单模态RGB图像、双模态RGB-Depth图像或三模态RGB-Depth-Thermal图像）。针对上述挑战，本文提出初步解决方案——模态切换网络（MSN）。具体而言，首先设计模态切换特征提取器（MSFE），通过引入模态指示因子为每种模态生成可切换权重的判别性特征；随后提出基于新型Transformer结构的动态融合模块（DFM），自适应融合可变数量模态的特征；最后构建新数据集AM-XD以推动AM SOD研究。大量实验表明，本方法能有效应对输入模态类型与数量的动态变化，实现鲁棒性显著性检测。