Training large-scale generative models is resource-intensive and relies heavily on heuristic dataset weighting. We address two fundamental questions: Can we train Large Language Models (LLMs) modularly-combining small, domain-specific experts to match monolithic performance-and can we do so robustly for any data mixture, eliminating heuristic tuning? We present a theoretical framework for modular generative modeling where a set of pre-trained experts are combined via a gating mechanism. We define the space of normalized gating functions, $G_{1}$, and formulate the problem as a minimax game to find a single robust gate that minimizes divergence to the worst-case data mixture. We prove the existence of such a robust gate using Kakutani's fixed-point theorem and show that modularity acts as a strong regularizer, with generalization bounds scaling with the lightweight gate's complexity. Furthermore, we prove that this modular approach can theoretically outperform models retrained on aggregate data, with the gap characterized by the Jensen-Shannon Divergence. Finally, we introduce a scalable Stochastic Primal-Dual algorithm and a Structural Distillation method for efficient inference. Empirical results on synthetic and real-world datasets confirm that our modular architecture effectively mitigates gradient conflict and can robustly outperform monolithic baselines.

翻译：训练大规模生成模型需要大量计算资源，且严重依赖启发式的数据集加权方法。我们针对两个基本问题展开研究：能否以模块化方式训练大语言模型（LLMs）——即通过组合小型领域专家模型来达到整体模型的性能水平？以及能否针对任意数据混合分布实现鲁棒训练，从而消除启发式调参？本文提出了一个模块化生成建模的理论框架，其中通过门控机制组合多个预训练的专家模型。我们定义了归一化门控函数空间 $G_{1}$，并将该问题构建为极小极大博弈，旨在寻找一个单一鲁棒门控函数，使其在最坏情况数据混合分布下的散度最小。利用角谷不动点定理，我们证明了此类鲁棒门控函数的存在性，并证明模块化本身是一种强正则化器，其泛化界随轻量级门控网络的复杂度而定。进一步，我们证明这种模块化方法在理论上能够优于在聚合数据上重新训练的模型，其性能差距可由Jensen-Shannon散度刻画。最后，我们提出了可扩展的随机原始-对偶算法以及用于高效推理的结构蒸馏方法。在合成数据集和真实数据集上的实验结果证实，我们的模块化架构能有效缓解梯度冲突，并能稳定超越整体式基线模型。