In this paper, we develop upon the topic of loss function learning, an emergent meta-learning paradigm that aims to learn loss functions that significantly improve the performance of the models trained under them. Specifically, we propose a new meta-learning framework for task and model-agnostic loss function learning via a hybrid search approach. The framework first uses genetic programming to find a set of symbolic loss functions. Second, the set of learned loss functions is subsequently parameterized and optimized via unrolled differentiation. The versatility and performance of the proposed framework are empirically validated on a diverse set of supervised learning tasks. Results show that the learned loss functions bring improved convergence, sample efficiency, and inference performance on tabulated, computer vision, and natural language processing problems, using a variety of task-specific neural network architectures.

翻译：本文在损失函数学习这一新兴元学习范式领域展开深入研究，旨在通过学习能够显著提升模型训练效果的损失函数。具体而言，我们提出了一种通过混合搜索方法实现任务与模型无关的损失函数学习的新型元学习框架。该框架首先利用遗传编程寻找一组符号化损失函数，随后通过展开微分对已学习的损失函数集进行参数化与优化。通过在多种监督学习任务上的实证验证，结果表明：所学习的损失函数在使用多种特定任务神经网络架构时，能够在表格数据、计算机视觉及自然语言处理问题上展现出更优的收敛性、样本效率及推理性能。