Large language models (LLMs) exhibit strengths across diverse domains. However, achieving strong performance across these domains with a single general-purpose model typically requires scaling to sizes that are prohibitively expensive to train and deploy. On the other hand, while smaller domain-specialized models are much more efficient, they struggle to generalize beyond their training distributions. To address this dilemma, we propose FusionRoute, a robust and effective token-level multi-LLM collaboration framework in which a lightweight router simultaneously (i) selects the most suitable expert at each decoding step and (ii) contributes a complementary logit that refines or corrects the selected expert's next-token distribution via logit addition. Unlike existing token-level collaboration methods that rely solely on fixed expert outputs, we provide a theoretical analysis showing that pure expert-only routing is fundamentally limited: unless strong global coverage assumptions hold, it cannot in general realize the optimal decoding policy. By augmenting expert selection with a trainable complementary generator, FusionRoute expands the effective policy class and enables recovery of optimal value functions under mild conditions. Empirically, across both Llama-3 and Gemma-2 families and diverse benchmarks spanning mathematical reasoning, code generation, and instruction following, FusionRoute outperforms both sequence- and token-level collaboration, model merging, and direct fine-tuning, while remaining competitive with domain experts on their respective tasks.

翻译：大型语言模型（LLM）在不同领域展现出各自的优势。然而，若要通过单一通用模型在所有领域均实现强劲性能，通常需要将模型规模扩展至训练与部署成本极高的程度。另一方面，尽管小型领域专用模型效率显著更高，但其难以泛化至训练分布之外。为解决这一困境，我们提出FusionRoute——一种鲁棒且有效的token级多LLM协作框架。该框架通过轻量级路由器同步实现以下功能：（1）在每个解码步骤选择最合适的专家模型；（2）生成互补logit，通过logit加法机制优化或校正所选专家的下一token分布。与现有仅依赖固定专家输出的token级协作方法不同，我们通过理论分析证明：纯专家路由机制存在根本性局限——除非满足强全局覆盖假设，否则其通常无法实现最优解码策略。FusionRoute通过将专家选择与可训练的互补生成器相结合，扩展了有效策略类别，并在温和条件下实现了最优价值函数的恢复。实证研究表明，在Llama-3与Gemma-2系列模型上，跨越数学推理、代码生成和指令遵循等多样化基准测试，FusionRoute在性能上均优于序列级与token级协作、模型融合及直接微调方法，同时在各自任务上保持与领域专家模型的竞争力。