The rapid pace at which new large language models (LLMs) appear, and older ones become obsolete, forces providers to manage a streaming inventory under a strict concurrency cap and per-query cost budgets. We cast this as an online decision problem that couples stage-wise deployment (at fixed maintenance windows) with per-query routing among live models. We introduce StageRoute, a hierarchical algorithm that (i) optimistically selects up to $M_{\max}$ models for the next stage using reward upper-confidence and cost lower-confidence bounds, and (ii) routes each incoming query by solving a budget- and throughput-constrained bandit subproblem over the deployed set. We prove a regret of $\tilde{\mathcal{O}}(T^{2/3})$ with a matching lower bound, establishing near-optimality, and validate the theory empirically: StageRoute tracks a strong oracle under tight budgets across diverse workloads.

翻译：新的大语言模型（LLM）快速涌现，而旧模型迅速过时，这迫使服务提供商必须在严格的并发上限和单次查询成本预算约束下管理流式模型库存。我们将此问题建模为一个在线决策问题，该问题将分阶段部署（在固定的维护窗口进行）与实时模型间的单次查询路由相耦合。本文提出StageRoute，一种分层算法，该算法（i）利用奖励上置信界与成本下置信界，以乐观方式为下一阶段选择最多$M_{\max}$个模型；（ii）通过在被部署的模型集合上求解一个预算与吞吐量约束的赌博机子问题，为每个到达的查询进行路由。我们证明了算法具有$\tilde{\mathcal{O}}(T^{2/3})$的遗憾上界，并给出了匹配的下界，从而确立了其近最优性。实证结果验证了理论分析：在多种工作负载下，即使预算紧张，StageRoute也能有效追踪一个强预言机基准的性能。