Large Language Models (LLMs) and other large foundation models have achieved noteworthy success, but their size exacerbates existing resource consumption and latency challenges. In particular, the large-scale deployment of these models is hindered by the significant resource requirements during inference. In this paper, we study two approaches for mitigating these challenges: employing a cache to store previous queries and learning a model multiplexer to choose from an ensemble of models for query processing. Theoretically, we provide an optimal algorithm for jointly optimizing both approaches to reduce the inference cost in both offline and online tabular settings. By combining a caching algorithm, namely Greedy Dual Size with Frequency (GDSF) or Least Expected Cost (LEC), with a model multiplexer, we achieve optimal rates in both offline and online settings. Empirically, simulations show that the combination of our caching and model multiplexing algorithms greatly improves over the baselines, with up to $50\times$ improvement over the baseline when the ratio between the maximum cost and minimum cost is $100$. Experiments on real datasets show a $4.3\times$ improvement in FLOPs over the baseline when the ratio for FLOPs is $10$, and a $1.8\times$ improvement in latency when the ratio for average latency is $1.85$.

翻译：大型语言模型及其他大型基础模型已取得显著成功，但其规模加剧了资源消耗与延迟挑战。具体而言，这些模型的大规模部署受限于推理过程中巨大的资源需求。本文研究两种缓解上述挑战的方法：采用缓存存储历史查询，以及学习一个模型复用器以从模型集成中选择处理查询的模型。理论上，我们提出一种联合优化这两种方法的最优算法，以降低离线与在线表格设定下的推理成本。通过结合GDSF（带频率的贪心双倍大小）或LEC（最小期望成本）等缓存算法与模型复用器，我们在离线与在线设定下均实现了最优速率。实验模拟表明，我们的缓存与模型复用算法组合相比基线方法有显著提升：当最大成本与最小成本之比为100时，改进幅度高达50倍。真实数据集上的实验显示，当FLOPs之比为10时，FLOPs较基线改进4.3倍；当平均延迟之比为1.85时，延迟改进1.8倍。