Mixture-of-experts (MoE) model incorporates the power of multiple submodels via gating functions to achieve greater performance in numerous regression and classification applications. From a theoretical perspective, while there have been previous attempts to comprehend the behavior of that model under the regression settings through the convergence analysis of maximum likelihood estimation in the Gaussian MoE model, such analysis under the setting of a classification problem has remained missing in the literature. We close this gap by establishing the convergence rates of density estimation and parameter estimation in the softmax gating multinomial logistic MoE model. Notably, when part of the expert parameters vanish, these rates are shown to be slower than polynomial rates owing to an inherent interaction between the softmax gating and expert functions via partial differential equations. To address this issue, we propose using a novel class of modified softmax gating functions which transform the input value before delivering them to the gating functions. As a result, the previous interaction disappears and the parameter estimation rates are significantly improved.

翻译：混合专家（MoE）模型通过门控函数整合多个子模型的能力，在众多回归与分类应用中实现了更优性能。从理论视角看，尽管已有研究尝试通过高斯MoE模型中极大似然估计的收敛性分析来理解该模型在回归场景下的行为，但在分类问题设定下的此类分析在文献中仍属空白。我们通过建立软最大门控多项逻辑MoE模型中密度估计与参数估计的收敛速度填补了这一空白。值得注意的是，当部分专家参数趋近于零时，由于软最大门控函数与专家函数通过偏微分方程产生的内在交互作用，这些收敛速度被证明慢于多项式速率。为解决该问题，我们提出一类新型修正软最大门控函数，该函数在将输入值传递至门控函数前对其进行变换。最终，原有交互作用被消除，参数估计速率获得显著提升。