Undeniably, Large Language Models (LLMs) have stirred an extraordinary wave of innovation in the machine learning research domain, resulting in substantial impact across diverse fields such as reinforcement learning, robotics, and computer vision. Their incorporation has been rapid and transformative, marking a significant paradigm shift in the field of machine learning research. However, the field of experimental design, grounded on black-box optimization, has been much less affected by such a paradigm shift, even though integrating LLMs with optimization presents a unique landscape ripe for exploration. In this position paper, we frame the field of black-box optimization around sequence-based foundation models and organize their relationship with previous literature. We discuss the most promising ways foundational language models can revolutionize optimization, which include harnessing the vast wealth of information encapsulated in free-form text to enrich task comprehension, utilizing highly flexible sequence models such as Transformers to engineer superior optimization strategies, and enhancing performance prediction over previously unseen search spaces.

翻译：毫无疑问，大型语言模型（LLMs）在机器学习研究领域掀起了空前的创新浪潮，对强化学习、机器人学和计算机视觉等多个领域产生了深远影响。它们的融入既迅速又具有变革性，标志着机器学习研究领域的重大范式转变。然而，基于黑箱优化的实验设计领域受这种范式转变的影响要小得多，尽管将LLMs与优化相结合为探索开辟了一片独特的天地。在这篇立场论文中，我们以基于序列的基础模型为框架定义黑箱优化领域，并梳理其与以往文献的关系。我们讨论了基础语言模型能够彻底改变优化的最有望途径，包括利用自由形式文本中蕴含的丰富信息来增强任务理解，运用Transformer等高度灵活的序列模型来设计更优的优化策略，以及在从未见过的搜索空间上改进性能预测。