Optimizing the deployment of Large language models (LLMs) is expensive today since it requires experimentally running an application workload against an LLM implementation while exploring large configuration space formed by system knobs such as parallelization strategies, batching techniques, and scheduling policies. To address this challenge, we present Vidur - a large-scale, high-fidelity, easily-extensible simulation framework for LLM inference performance. Vidur models the performance of LLM operators using a combination of experimental profiling and predictive modeling, and evaluates the end-to-end inference performance for different workloads by estimating several metrics of interest such as latency and throughput. We validate the fidelity of Vidur on several LLMs and show that it estimates inference latency with less than 9% error across the range. Further, we present Vidur-Search, a configuration search tool that helps optimize LLM deployment. Vidur-Search uses Vidur to automatically identify the most cost-effective deployment configuration that meets application performance constraints. For example, Vidur-Search finds the best deployment configuration for LLaMA2-70B in one hour on a CPU machine, in contrast to a deployment-based exploration which would require 42K GPU hours - costing ~218K dollars. Source code for Vidur is available at https://github.com/microsoft/vidur.

翻译：优化大语言模型（LLM）的部署目前成本高昂，因为需要在系统参数（如并行策略、批处理技术和调度策略）构成的大配置空间中，通过实验方式运行应用程序工作负载以测试LLM实现。为解决这一挑战，我们提出Vidur——一个大规模、高保真、易扩展的LLM推理性能仿真框架。Vidur结合实验剖析与预测建模对LLM算子性能进行建模，并通过估计延迟、吞吐量等多项指标评估不同工作负载的端到端推理性能。我们在多个LLM上验证了Vidur的保真度，结果表明其对推理延迟的估计误差低于9%。此外，我们提出配置搜索工具Vidur-Search，用于优化LLM部署。Vidur-Search借助Vidur自动识别满足应用性能约束的最经济部署配置。例如，Vidur-Search在单台CPU机器上仅需1小时即可找到LLaMA2-70B的最佳部署配置，而基于实际部署的探索需要42,000 GPU小时（约耗费21.8万美元）。Vidur源代码已发布于 https://github.com/microsoft/vidur。