Mixture-of-experts networks (MoEs) have demonstrated remarkable efficiency in modern deep learning. Despite their empirical success, the theoretical foundations underlying their ability to model complex tasks remain poorly understood. In this work, we conduct a systematic study of the expressive power of MoEs in modeling complex tasks with two common structural priors: low-dimensionality and sparsity. For shallow MoEs, we prove that they can efficiently approximate functions supported on low-dimensional manifolds, overcoming the curse of dimensionality. For deep MoEs, we show that $\mathcal{O}(L)$-layer MoEs with $E$ experts per layer can approximate piecewise functions comprising $E^L$ pieces with compositional sparsity, i.e., they can exhibit an exponential number of structured tasks. Our analysis reveals the roles of critical architectural components and hyperparameters in MoEs, including the gating mechanism, expert networks, the number of experts, and the number of layers, and offers natural suggestions for MoE variants.

翻译：混合专家网络（MoEs）在现代深度学习中展现出卓越的效率。尽管其经验性成功显著，但支撑其建模复杂任务能力的理论基础仍不甚明晰。本研究系统性地探讨了MoEs在建模具有两种常见结构先验（低维性与稀疏性）的复杂任务时的表达能力。对于浅层MoEs，我们证明其能高效逼近支撑在低维流形上的函数，从而克服维度灾难。对于深层MoEs，我们表明具有每层$E$个专家的$\mathcal{O}(L)$层MoEs能够逼近包含$E^L$个分段且具有组合稀疏性的分段函数，即它们能呈现指数级数量的结构化任务。我们的分析揭示了MoEs中关键架构组件与超参数的作用，包括门控机制、专家网络、专家数量及网络层数，并为MoE变体提供了自然的改进建议。