Large language models (LLMs) propel the prosperity of interactive AI applications showcased by ChatGPT that demand timely response of inference services. However, LLM inference is computation intensive and memory intensive, and improper parameter configuration at LLM platforms may exacerbate the inference time. In this paper, we analyze the impact of LLM output token distribution on the inference queueing delay, where the max-token clipping and the batched inference are considered. By formulating an M/G/1 model, we observe that enforcing a maximum output token limit on a very small fraction of inference requests can significantly reduce the queueing delay, and our model facilitates the selection of the optimal limit. For the batch inference, we model the service process as a bulk queue in which the batch processing time is affected by the batch size and the maximum token size inside this batch jointly. The queueing delays of the batching of all buffered requests (dynamic batching), the batching of constant number of requests (fixed batching), and the batching without intra-batch waiting (elastic batching) are derived. Experimental results show that our mathematical models coincide with the event-driven simulations well.

翻译：大型语言模型（LLM）推动了以ChatGPT为代表的交互式AI应用的繁荣，这类应用对推理服务的实时响应提出了更高要求。然而，LLM推理具有计算密集与内存密集的特性，且LLM平台中不当的参数配置可能进一步加剧推理耗时。本文分析了LLM输出令牌分布对推理排队延迟的影响，其中考虑了最大令牌截断与批量推理机制。通过建立M/G/1排队模型，我们发现仅对极少部分推理请求施加最大输出令牌限制即可显著降低排队延迟，且该模型为最优限制值的选取提供了理论依据。针对批量推理场景，我们将服务过程建模为批量队列，其中批处理时间同时受批次大小与该批次内最大令牌数共同影响。本文推导了全缓冲请求批量处理（动态批处理）、固定请求数批量处理（静态批处理）以及无批内等待的批量处理（弹性批处理）三种策略下的排队延迟。实验结果表明，我们的数学模型与事件驱动仿真结果高度吻合。