Automatic performance tuning (auto-tuning) is essential for optimizing high-performance applications, where vast and irregular search spaces make manual exploration infeasible. While auto-tuners traditionally rely on classical approaches such as evolutionary, annealing, or surrogate-based optimizers, designing algorithms that efficiently find near-optimal configurations robustly across diverse tasks is challenging. We propose a new paradigm: using large language models (LLMs) to automatically generate optimization algorithms tailored to auto-tuning problems. We introduce a framework that prompts LLMs with problem descriptions and search space characteristics to synthesize, test, and iteratively refine specialized optimizers. These generated algorithms are evaluated on four real-world auto-tuning applications across six hardware platforms and compared against the state-of-the-art in two contemporary auto-tuning frameworks. The evaluation demonstrates that providing additional application- and search space-specific information in the generation stage results in an average performance improvement of 30.7% and 14.6%, respectively. In addition, our results show that LLM-generated optimizers can rival, and in various cases outperform, existing human-designed algorithms, with our best-performing generated optimization algorithms achieving an average 72.4% improvement over state-of-the-art optimizers for auto-tuning.

翻译：自动性能调优（auto-tuning）对于优化高性能应用程序至关重要，其中庞大且不规则的搜索空间使得手动探索不可行。尽管自动调优器传统上依赖进化算法、退火算法或基于代理的优化器等经典方法，但设计能够稳健地在不同任务中高效找到近优配置的算法仍具挑战性。我们提出了一种新范式：利用大型语言模型（LLMs）自动生成针对自动调优问题定制的优化算法。我们引入了一个框架，该框架通过问题描述和搜索空间特征提示LLMs，以合成、测试并迭代优化专用优化器。这些生成的算法在跨越六个硬件平台的四个真实自动调优应用上进行了评估，并与两个当代自动调优框架中的现有最优技术进行了比较。评估表明，在生成阶段提供额外的应用与搜索空间特定信息，分别带来了平均30.7%和14.6%的性能提升。此外，我们的结果显示，LLM生成的优化器能够与现有的人工设计算法相匹敌，并在多种情况下超越它们，其中我们性能最佳的生成优化算法在自动调优方面比现有最优优化器实现了平均72.4%的提升。