Modern GPU workloads, especially large language model (LLM) inference, suffer from kernel launch overheads and coarse synchronization that limit inter-kernel parallelism. Recent megakernel techniques fuse multiple operators into a single persistent kernel to eliminate launch gaps and expose inter-kernel parallelism, but struggle to handle dynamic shapes and data-dependent computation in real workloads. We present Event Tensor, a unified compiler abstraction for dynamic megakernels. Event Tensor encodes dependencies between tiled tasks, and enables first-class support for both shape and data-dependent dynamism. Built atop this abstraction, our Event Tensor Compiler (ETC) applies static and dynamic scheduling transformations to generate high-performance persistent kernels. Evaluations show that ETC achieves state-of-the-art LLM serving latency while significantly reducing system warmup overhead.

翻译：现代GPU工作负载，尤其是大型语言模型（LLM）推理，受限于内核启动开销和粗粒度同步，限制了内核间的并行性。近期的大内核技术将多个算子融合为单个持久内核，以消除启动间隙并暴露内核间并行性，但在处理实际工作负载中的动态形状和数据依赖计算时面临挑战。我们提出事件张量（Event Tensor），一种面向动态大内核的统一编译器抽象。事件张量对分块任务间的依赖关系进行编码，并首次为形状依赖和数据依赖的动态性提供原生支持。基于此抽象，我们构建的事件张量编译器（ETC）应用静态和动态调度变换，生成高性能持久内核。评估表明，ETC在实现最先进LLM服务延迟的同时，显著降低了系统预热开销。