Mixture of experts (MoE) model is a statistical machine learning design that aggregates multiple expert networks using a softmax gating function in order to form a more intricate and expressive model. Despite being commonly used in several applications owing to their scalability, the mathematical and statistical properties of MoE models are complex and difficult to analyze. As a result, previous theoretical works have primarily focused on probabilistic MoE models by imposing the impractical assumption that the data are generated from a Gaussian MoE model. In this work, we investigate the performance of the least squares estimators (LSE) under a deterministic MoE model where the data are sampled according to a regression model, a setting that has remained largely unexplored. We establish a condition called strong identifiability to characterize the convergence behavior of various types of expert functions. We demonstrate that the rates for estimating strongly identifiable experts, namely the widely used feed forward networks with activation functions $\mathrm{sigmoid}(\cdot)$ and $\tanh(\cdot)$, are substantially faster than those of polynomial experts, which we show to exhibit a surprising slow estimation rate. Our findings have important practical implications for expert selection.

翻译：混合专家模型是一种统计机器学习设计，通过使用softmax门控函数聚合多个专家网络，以形成更复杂且更具表达力的模型。尽管因可扩展性而在多种应用中广泛使用，但混合专家模型的数学和统计特性复杂且难以分析。因此，以往的理论工作主要集中于概率混合专家模型，这些研究施加了不切实际的假设，即数据由高斯混合专家模型生成。在本工作中，我们研究了数据依据回归模型采样（这一设定迄今尚未被充分探索）的确定性混合专家模型下最小二乘估计的性能。我们提出一个称为强可辨识性的条件，以刻画各类专家函数的收敛行为。我们证明：能实现强可辨识的专家（即广泛使用的激活函数为$\mathrm{sigmoid}(\cdot)$和$\tanh(\cdot)$的前馈网络）的估计速率显著快于多项式专家——后者的估计速率呈现出令人惊讶的缓慢特性。我们的发现对专家选择具有重要的实践意义。