Large Language Models (LLMs) have achieved strong performance across natural language and multimodal tasks, yet their practical deployment remains constrained by inference latency and kernel launch overhead, particularly in interactive, short-sequence settings. This paper presents a hybrid runtime framework that combines Just-In-Time (JIT) compilation with CUDA Graph execution to reduce launch overhead while preserving runtime flexibility during autoregressive decoding. The framework partitions transformer inference into static components executed via CUDA Graph replay and dynamic components handled through JIT-compiled kernels, enabling asynchronous graph capture and reuse across decoding steps. We evaluate the proposed approach on LLaMA-2 7B using single-GPU, batch-size-one inference across prompt lengths from 10 to 500 tokens. Experimental results show that the hybrid runtime reduces Time-to-First-Token (TTFT) by up to 66.0% and achieves lower P99 latency compared with TensorRT-LLM in this regime. These results indicate that hybrid JIT-CUDA Graph execution can effectively reduce inference latency and variance for short-sequence LLM workloads, making it a practical optimization strategy for latency-sensitive AI applications.

翻译：大型语言模型（LLM）在自然语言和多模态任务中取得了优异性能，但其实际部署仍受限于推理延迟和内核启动开销，特别是在交互式短序列场景中。本文提出一种混合运行时框架，结合即时（JIT）编译与CUDA图执行，在自回归解码过程中降低启动开销的同时保持运行时灵活性。该框架将Transformer推理划分为通过CUDA图回放执行的静态组件和由JIT编译内核处理的动态组件，从而实现异步图捕获并跨解码步骤重用。我们采用单GPU、批量大小为1的推理配置，在提示长度从10到500个token范围内，对LLaMA-2 7B模型评估所提方法。实验结果表明，在该场景下，混合运行时相比TensorRT-LLM能将首token延迟（TTFT）降低高达66.0%，并实现更低的P99延迟。这些结果证明，混合JIT-CUDA图执行能有效降低短序列LLM工作负载的推理延迟和方差，从而成为延迟敏感型AI应用的一种实用优化策略。