As Generative AI (GenAI), particularly inference, rapidly emerges as a dominant workload category, the Kubernetes ecosystem is proactively evolving to natively support its unique demands. This industry paper demonstrates how emerging Kubernetes-native projects can be combined to deliver the benefits of container orchestration, such as scalability and resource efficiency, to complex AI workflows. We implement and evaluate an illustrative, multi-stage use case consisting of automatic speech recognition and summarization. First, we address batch inference by using Kueue to manage jobs that transcribe audio files with Whisper models and Dynamic Accelerator Slicer (DAS) to increase parallel job execution. Second, we address a discrete online inference scenario by feeding the transcripts to a Large Language Model for summarization hosted using llm-d, a novel solution utilizing the recent developments around the Kubernetes Gateway API Inference Extension (GAIE) for optimized routing of inference requests. Our findings illustrate that these complementary components (Kueue, DAS, and GAIE) form a cohesive, high-performance platform, proving Kubernetes' capability to serve as a unified foundation for demanding GenAI workloads: Kueue reduced total makespan by up to 15%; DAS shortened mean job completion time by 36\%; and GAIE working in conjunction with llm-d improved tail Time to First Token latency by up to 90% even under high loads.

翻译：随着生成式人工智能（GenAI）特别是推理任务迅速崛起成为主流工作负载类别，Kubernetes生态系统正积极演进以原生支持其独特需求。本产业技术论文展示了如何将新兴的Kubernetes原生项目组合运用，从而为复杂AI工作流提供容器编排的可扩展性和资源效率优势。我们实现并评估了一个包含自动语音识别与摘要生成的多阶段示例用例。首先，我们通过Kueue管理使用Whisper模型转录音频文件的批处理推理任务，并采用动态加速器切片器（DAS）提升并行作业执行效率。其次，我们构建离散在线推理场景：将转录文本输入大型语言模型进行摘要生成，该模型通过llm-d方案部署——这是一种利用Kubernetes网关API推理扩展（GAIE）最新进展实现推理请求优化路由的创新解决方案。实验结果表明，这些互补组件（Kueue、DAS与GAIE）共同构成了协调统一的高性能平台，证实了Kubernetes能够作为高要求GenAI工作负载的统一基础架构：Kueue将总完工时间缩短达15%；DAS使平均作业完成时间减少36%；GAIE与llm-d协同工作，即使在高负载下也能将尾部首令牌延迟降低高达90%。