Optimizing Large Language Model (LLM) inference in production systems is increasingly difficult due to dynamic workloads, stringent latency/throughput targets, and a rapidly expanding configuration space. This complexity spans not only distributed parallelism strategies (tensor/pipeline/expert) but also intricate framework-specific runtime parameters such as those concerning the enablement of CUDA graphs, available KV-cache memory fractions, and maximum token capacity, which drastically impact performance. The diversity of modern inference frameworks (e.g., TRT-LLM, vLLM, SGLang), each employing distinct kernels and execution policies, makes manual tuning both framework-specific and computationally prohibitive. We present AIConfigurator, a unified performance-modeling system that enables rapid, framework-agnostic inference configuration search without requiring GPU-based profiling. AIConfigurator combines (1) a methodology that decomposes inference into analytically modelable primitives - GEMM, attention, communication, and memory operations while capturing framework-specific scheduling dynamics; (2) a calibrated kernel-level performance database for these primitives across a wide range of hardware platforms and popular open-weights models (GPT-OSS, Qwen, DeepSeek, LLama, Mistral); and (3) an abstraction layer that automatically resolves optimal launch parameters for the target backend, seamlessly integrating into production-grade orchestration systems. Evaluation on production LLM serving workloads demonstrates that AIConfigurator identifies superior serving configurations that improve performance by up to 40% for dense models (e.g., Qwen3-32B) and 50% for MoE architectures (e.g., DeepSeek-V3), while completing searches within 30 seconds on average. Enabling the rapid exploration of vast design spaces - from cluster topology down to engine specific flags.

翻译：在生产系统中优化大语言模型推理正变得日益困难，原因在于动态的工作负载、严格的延迟/吞吐量目标以及快速扩张的配置空间。这种复杂性不仅涉及分布式并行策略（张量/流水线/专家并行），还包括复杂的框架特定运行时参数，例如关于启用CUDA图、可用KV缓存内存比例以及最大令牌容量的参数，这些参数会极大地影响性能。现代推理框架（如TRT-LLM、vLLM、SGLang）的多样性，各自采用不同的内核和执行策略，使得手动调优既具有框架特异性，又在计算上代价高昂。我们提出了AIConfigurator，一个统一的性能建模系统，它支持快速、与框架无关的推理配置搜索，而无需基于GPU的性能剖析。AIConfigurator结合了以下三点：(1) 一种将推理分解为可分析建模的原语（GEMM、注意力、通信和内存操作）并捕获框架特定调度动态的方法论；(2) 一个针对这些原语、跨广泛硬件平台和流行开源模型（GPT-OSS、Qwen、DeepSeek、LLama、Mistral）的校准内核级性能数据库；(3) 一个抽象层，能自动为目标后端解析最优启动参数，并无缝集成到生产级编排系统中。在生产LLM服务负载上的评估表明，AIConfigurator能识别出更优的服务配置，对于稠密模型（如Qwen3-32B）性能提升高达40%，对于MoE架构（如DeepSeek-V3）提升高达50%，同时平均在30秒内完成搜索。该系统实现了从集群拓扑到引擎特定标志的广阔设计空间的快速探索。