Large Language Models (LLMs) and other large foundation models have achieved noteworthy success, but their size exacerbates existing resource consumption and latency challenges. In particular, the large-scale deployment of these models is hindered by the significant resource requirements during inference. In this paper, we study two approaches for mitigating these challenges: employing a cache to store previous queries and learning a model multiplexer to choose from an ensemble of models for query processing. Theoretically, we provide an optimal algorithm for jointly optimizing both approaches to reduce the inference cost in both offline and online tabular settings. By combining a caching algorithm, namely Greedy Dual Size with Frequency (GDSF) or Least Expected Cost (LEC), with a model multiplexer, we achieve optimal rates in both offline and online settings. Empirically, simulations show that the combination of our caching and model multiplexing algorithms greatly improves over the baselines, with up to $50\times$ improvement over the baseline when the ratio between the maximum cost and minimum cost is $100$. Experiments on real datasets show a $4.3\times$ improvement in FLOPs over the baseline when the ratio for FLOPs is $10$, and a $1.8\times$ improvement in latency when the ratio for average latency is $1.85$.

翻译：大型语言模型（LLMs）及其他大型基础模型已取得显著成功，但其规模加剧了资源消耗和延迟挑战。特别是，这些模型的大规模部署受到推理过程中巨大资源需求的阻碍。本文研究了缓解这些挑战的两种方法：使用缓存存储先前查询，以及学习模型复用器以从模型集合中选择处理查询的模型。理论上，我们提出了一种联合优化这两种方法的最优算法，以降低离线与在线表格场景下的推理成本。通过将缓存算法（即贪心双重大小频率算法GDSF或最小期望成本算法LEC）与模型复用器结合，我们在离线与在线设置中均实现了最优速率。实证上，仿真表明我们的缓存与模型复用算法组合相较于基线有显著提升：当最大成本与最小成本之比为100时，性能提升高达50倍。在真实数据集实验中，当FLOPs比值为10时，FLOPs较基线提升4.3倍；当平均延迟比值为1.85时，延迟较基线提升1.8倍。