Large language models (LLMs) exhibit strengths across diverse domains. However, achieving strong performance across these domains with a single general-purpose model typically requires scaling to sizes that are prohibitively expensive to train and deploy. On the other hand, while smaller domain-specialized models are much more efficient, they struggle to generalize beyond their training distributions. To address this dilemma, we propose FusionRoute, a robust and effective token-level multi-LLM collaboration framework in which a lightweight router simultaneously (i) selects the most suitable expert at each decoding step and (ii) contributes a complementary logit that refines or corrects the selected expert's next-token distribution via logit addition. Unlike existing token-level collaboration methods that rely solely on fixed expert outputs, we provide a theoretical analysis showing that pure expert-only routing is fundamentally limited: unless strong global coverage assumptions hold, it cannot in general realize the optimal decoding policy. By augmenting expert selection with a trainable complementary generator, FusionRoute expands the effective policy class and enables recovery of optimal value functions under mild conditions. Empirically, across both Llama-3 and Gemma-2 families and diverse benchmarks spanning mathematical reasoning, code generation, and instruction following, FusionRoute outperforms both sequence- and token-level collaboration, model merging, and direct fine-tuning, while remaining competitive with domain experts on their respective tasks.

翻译：大型语言模型在不同领域展现出优势，但使用单一通用模型在这些领域取得优异表现通常需要扩展至训练和部署成本极高的规模。相比之下，较小的领域专用模型虽然效率更高，却难以泛化至训练分布之外。为解决这一困境，我们提出FusionRoute——一个鲁棒且高效的token级多模型协作框架。该框架通过轻量级路由器同时实现：(i) 在每个解码步骤选择最合适的专家模型；(ii) 贡献互补logits，通过logit加法修正所选专家的下一token分布。不同于现有仅依赖固定专家输出的token级协作方法，我们通过理论分析证明纯专家路由存在根本性局限：除非满足强全局覆盖假设，否则通常无法实现最优解码策略。FusionRoute通过可训练互补生成器增强专家选择机制，在温和条件下扩展有效策略类别并实现最优价值函数的恢复。实验表明，在Llama-3和Gemma-2系列模型、涵盖数学推理、代码生成与指令遵循的多样化基准测试中，FusionRoute优于序列级与token级协作、模型融合及直接微调方法，同时在各自任务上与领域专家模型保持竞争力。