We explore the novel application of Large Language Models to code optimization. We present a 7B-parameter transformer model trained from scratch to optimize LLVM assembly for code size. The model takes as input unoptimized assembly and outputs a list of compiler options to best optimize the program. Crucially, during training, we ask the model to predict the instruction counts before and after optimization, and the optimized code itself. These auxiliary learning tasks significantly improve the optimization performance of the model and improve the model's depth of understanding. We evaluate on a large suite of test programs. Our approach achieves a 3.0% improvement in reducing instruction counts over the compiler, outperforming two state-of-the-art baselines that require thousands of compilations. Furthermore, the model shows surprisingly strong code reasoning abilities, generating compilable code 91% of the time and perfectly emulating the output of the compiler 70% of the time.

翻译：我们探索了大型语言模型在代码优化中的新颖应用。我们提出一个从头训练的7B参数Transformer模型，用于优化LLVM汇编代码的尺寸。该模型以未优化的汇编代码作为输入，输出一系列编译器选项以实现程序的最佳优化。关键的是，在训练过程中，我们要求模型预测优化前后的指令计数以及优化后的代码本身。这些辅助学习任务显著提升了模型的优化性能，并增强了模型的理解深度。我们在一个大型测试程序套件上进行了评估。我们的方法在降低指令计数方面比编译器提升了3.0%，优于两个需要数千次编译的先进基线方法。此外，该模型展现出令人惊讶的强大代码推理能力：91%的情况下能生成可编译代码，70%的情况下能完美模拟编译器的输出。