Recent advances in mixture-of-experts architectures have shown that individual experts models can be trained federatedly, i.e., in isolation from other experts by using a common base model to facilitate coordination. However, we hypothesize that full-sized experts may not be necessary for all domains and that instead low-rank adapters may be sufficient. Here, we introduce FlexMoRE, a Flexible Mixture of Rank-heterogenous Experts, which may be either full-sized experts or adapters of a suitable rank. We systematically investigate the trade-off between expert rank and downstream task performance by evaluating $6$ experts with ranks $2^0$ to $2^{14}$ resulting in experiments covering 150 mixtures (96 with 2 experts, 54 with 7 experts) that are evaluated across $120$ tasks. For our experiments, we build on FlexOlmo and turn its pre-trained experts into low-rank versions. Our regression analysis from expert rank to downstream task performance reveals that the best-performing rank is substantially higher for reasoning-heavy benchmarks than for knowledge-heavy benchmarks. These findings on rank sensitivity come with direct implications for memory efficiency: Using optimal ranks, FlexMoRE yields improved downstream task performance (average score $47.18$) compared to the baseline FlexOlmo-style mixture of full-sized experts (average score $45.46$) at less than one third the parameters ($10.75$B for FlexMoRE vs. $33.27$B for FlexOlmo). All code will be made available.

翻译：近年来，专家混合架构的研究进展表明，可以通过使用一个共享的基础模型来协调，以联邦方式（即与其他专家隔离地）训练各个专家模型。然而，我们假设对于所有领域而言，完整规模的专家可能并非必要，而低秩适配器可能就足够了。本文提出FlexMoRE，一种灵活的秩异构专家混合架构，其中的专家可以是完整规模的专家，也可以是具有适当秩的适配器。我们通过评估秩从 $2^0$ 到 $2^{14}$ 的 $6$ 位专家，系统地研究了专家秩与下游任务性能之间的权衡，实验覆盖了150种混合配置（96种含2位专家，54种含7位专家），并在 $120$ 项任务上进行了评估。在我们的实验中，我们基于FlexOlmo，并将其预训练的专家转换为低秩版本。从专家秩到下游任务性能的回归分析表明，在推理密集型基准测试中，最佳性能对应的秩显著高于知识密集型基准测试。这些关于秩敏感性的发现对内存效率具有直接意义：使用最优秩时，FlexMoRE在下游任务性能（平均得分 $47.18$）上优于基线FlexOlmo风格的完整规模专家混合模型（平均得分 $45.46$），且参数量不到其三分之一（FlexMoRE为 $10.75$B，而FlexOlmo为 $33.27$B）。所有代码将公开提供。