Sparsely-gated Mixture of Expert (MoE) layers have been recently successfully applied for scaling large transformers, especially for language modeling tasks. An intriguing side effect of sparse MoE layers is that they convey inherent interpretability to a model via natural expert specialization. In this work, we apply sparse MoE layers to CNNs for computer vision tasks and analyze the resulting effect on model interpretability. To stabilize MoE training, we present both soft and hard constraint-based approaches. With hard constraints, the weights of certain experts are allowed to become zero, while soft constraints balance the contribution of experts with an additional auxiliary loss. As a result, soft constraints handle expert utilization better and support the expert specialization process, while hard constraints maintain more generalized experts and increase overall model performance. Our findings demonstrate that experts can implicitly focus on individual sub-domains of the input space. For example, experts trained for CIFAR-100 image classification specialize in recognizing different domains such as flowers or animals without previous data clustering. Experiments with RetinaNet and the COCO dataset further indicate that object detection experts can also specialize in detecting objects of distinct sizes.

翻译：稀疏门控混合专家（MoE）层最近被成功应用于扩展大型Transformer，尤其是在语言建模任务中。稀疏MoE层的一个有趣副作用是，它们通过自然的专家专业化机制为模型带来了内在的可解释性。在本工作中，我们将稀疏MoE层应用于计算机视觉任务的CNN，并分析其对模型可解释性的影响。为稳定MoE训练，我们提出了基于软约束和硬约束两种方法。在硬约束下，某些专家的权重可被设为零，而软约束则通过额外的辅助损失平衡专家的贡献。结果表明，软约束能更好地处理专家利用率并支持专家专业化过程，而硬约束则保留了更通用的专家并提升了模型整体性能。我们的发现表明，专家可隐式聚焦于输入空间的独立子域。例如，针对CIFAR-100图像分类训练的专家，无需预先数据聚类即可专门识别花卉或动物等不同领域。基于RetinaNet和COCO数据集的实验进一步表明，目标检测专家还可专门检测不同尺寸的物体。