Large language models (LLMs) exhibit strengths across diverse domains. However, achieving strong performance across these domains with a single general-purpose model typically requires scaling to sizes that are prohibitively expensive to train and deploy. On the other hand, while smaller domain-specialized models are much more efficient, they struggle to generalize beyond their training distributions. To address this dilemma, we propose FusionRoute, a robust and effective token-level multi-LLM collaboration framework in which a lightweight router simultaneously (i) selects the most suitable expert at each decoding step and (ii) contributes a complementary logit that refines or corrects the selected expert's next-token distribution via logit addition. Unlike existing token-level collaboration methods that rely solely on fixed expert outputs, we provide a theoretical analysis showing that pure expert-only routing is fundamentally limited: unless strong global coverage assumptions hold, it cannot in general realize the optimal decoding policy. By augmenting expert selection with a trainable complementary generator, FusionRoute expands the effective policy class and enables recovery of optimal value functions under mild conditions. Empirically, across both Llama-3 and Gemma-2 families and diverse benchmarks spanning mathematical reasoning, code generation, and instruction following, FusionRoute outperforms both sequence- and token-level collaboration, model merging, and direct fine-tuning, while remaining competitive with domain experts on their respective tasks.

翻译：大语言模型（LLM）在不同领域展现出各自的优势。然而，要构建一个在所有这些领域均表现优异的通用模型，通常需要将模型规模扩展至训练和部署成本极高的程度。另一方面，虽然针对特定领域的小型模型效率更高，但它们难以泛化到训练分布之外的数据。为解决这一困境，我们提出FusionRoute——一个鲁棒且有效的令牌级多LLM协作框架。该框架通过一个轻量级路由器同时实现以下功能：（i）在每个解码步骤选择最合适的专家模型；（ii）生成互补对数，通过对数加法来优化或修正所选专家模型的下一个令牌分布。与现有仅依赖固定专家输出的令牌级协作方法不同，我们通过理论分析表明，纯专家路由机制存在根本性局限性：除非满足强全局覆盖假设，否则通常无法实现最优解码策略。FusionRoute通过引入可训练的互补生成器来增强专家选择机制，从而扩展了有效策略类别，并在温和条件下实现了最优价值函数的恢复。实证研究表明，在Llama-3和Gemma-2系列模型上，跨越数学推理、代码生成和指令遵循等多个基准测试，FusionRoute在性能上均优于序列级与令牌级协作、模型融合及直接微调方法，同时在各自任务上保持与领域专家模型的竞争力。