In this paper, we develop upon the emerging topic of loss function learning, which aims to learn loss functions that significantly improve the performance of the models trained under them. Specifically, we propose a new meta-learning framework for learning model-agnostic loss functions via a hybrid neuro-symbolic search approach. The framework first uses evolution-based methods to search the space of primitive mathematical operations to find a set of symbolic loss functions. Second, the set of learned loss functions are subsequently parameterized and optimized via an end-to-end gradient-based training procedure. The versatility of the proposed framework is empirically validated on a diverse set of supervised learning tasks. Results show that the meta-learned loss functions discovered by the newly proposed method outperform both the cross-entropy loss and state-of-the-art loss function learning methods on a diverse range of neural network architectures and datasets.

翻译：本文针对损失函数学习这一新兴课题展开研究，该课题旨在通过学习得到的损失函数显著提升在其指导下训练模型的性能。具体而言，我们提出了一种新的元学习框架，通过混合神经符号搜索方法学习模型无关的损失函数。该框架首先采用基于进化的方法在原始数学运算空间中搜索，以发现一组符号化损失函数。随后，将习得的损失函数集合进行参数化，并通过端到端的基于梯度的训练流程进行优化。我们在多种监督学习任务上实证验证了所提框架的通用性。实验结果表明，新方法发现的元学习损失函数在多样化的神经网络架构和数据集上，其性能均优于交叉熵损失函数以及当前最先进的损失函数学习方法。