In this paper, we develop upon the emerging topic of loss function learning, which aims to learn loss functions that significantly improve the performance of the models trained under them. Specifically, we propose a new meta-learning framework for learning model-agnostic loss functions via a hybrid neuro-symbolic search approach. The framework first uses evolution-based methods to search the space of primitive mathematical operations to find a set of symbolic loss functions. Second, the set of learned loss functions are subsequently parameterized and optimized via an end-to-end gradient-based training procedure. The versatility of the proposed framework is empirically validated on a diverse set of supervised learning tasks. Results show that the meta-learned loss functions discovered by the newly proposed method outperform both the cross-entropy loss and state-of-the-art loss function learning methods on a diverse range of neural network architectures and datasets.

翻译：本文围绕损失函数学习这一新兴课题展开研究，旨在学习能显著提升模型性能的损失函数。具体而言，我们提出了一种新的元学习框架，通过混合神经符号搜索方法学习模型无关的损失函数。该框架首先采用基于进化的方法搜索基本数学运算空间，以发现一组符号化损失函数；随后，这些学习得到的损失函数通过端到端梯度训练过程进行参数化与优化。通过在多种监督学习任务上的实证验证，结果表明：新方法发现的元学习损失函数在多种神经网络架构和数据集上均优于交叉熵损失函数及当前最先进的损失函数学习方法。