As agentic AI systems become increasingly capable of generating and optimizing GPU kernels, progress is constrained by benchmarks that reward speedup over software baselines rather than proximity to hardware-efficient execution. We present SOL-ExecBench, a benchmark of 235 CUDA kernel optimization problems extracted from 124 production and emerging AI models spanning language, diffusion, vision, audio, video, and hybrid architectures, targeting NVIDIA Blackwell GPUs. The benchmark covers forward and backward workloads across BF16, FP8, and NVFP4, including kernels whose best performance is expected to rely on Blackwell-specific capabilities. Unlike prior benchmarks that evaluate kernels primarily relative to software implementations, SOL-ExecBench measures performance against analytically derived Speed-of-Light (SOL) bounds computed by SOLAR, our pipeline for deriving hardware-grounded SOL bounds, yielding a fixed target for hardware-efficient optimization. We report a SOL Score that quantifies how much of the gap between a release-defined scoring baseline and the hardware SOL bound a candidate kernel closes. To support robust evaluation of agentic optimizers, we additionally provide a sandboxed harness with GPU clock locking, L2 cache clearing, isolated subprocess execution, and static analysis based checks against common reward-hacking strategies. SOL-ExecBench reframes GPU kernel benchmarking from beating a mutable software baseline to closing the remaining gap to hardware Speed-of-Light.

翻译：随着智能体AI系统生成和优化GPU内核的能力日益增强，现有基准测试因侧重于比较软件基线加速比而非逼近硬件高效执行而受到约束。我们提出SOL-ExecBench，这是一个包含235个CUDA内核优化问题的基准测试，这些问题源自横跨语言、扩散、视觉、音频、视频及混合架构的124个生产级和新兴AI模型，目标硬件为NVIDIA Blackwell GPU。该基准测试涵盖BF16、FP8和NVFP4数据格式的前向与反向工作负载，其中部分内核的最佳性能预期需依赖Blackwell特有能力。与先前主要基于软件实现评估内核的基准不同，SOL-ExecBench通过衡量由SOLAR（我们的硬件基态光速边界推导流水线）解析计算得到的理论光速（SOL）边界来评估性能，从而为硬件高效优化提供固定目标。我们提出的SOL评分量化了候选内核在释放定义评分基线与硬件SOL边界之间所缩小的差距。为支持对智能体优化器的稳健评估，我们还提供了沙箱化测试框架，具备GPU时钟锁定、L2缓存清除、隔离子进程执行以及针对常见奖励欺骗策略的静态分析检查。SOL-ExecBench将GPU内核基准测试从“超越可变软件基线”重新定义为“逼近硬件光速极限”。