All machine learning algorithms use a loss, cost, utility or reward function to encode the learning objective and oversee the learning process. This function that supervises learning is a frequently unrecognized hyperparameter that determines how incorrect outputs are penalized and can be tuned to improve performance. This paper shows that training speed and final accuracy of neural networks can significantly depend on the loss function used to train neural networks. In particular derivative values can be significantly different with different loss functions leading to significantly different performance after gradient descent based Backpropagation (BP) training. This paper explores the effect on performance of new loss functions that are more liberal or strict compared to the popular Cross-entropy loss in penalizing incorrect outputs. Eight new loss functions are proposed and a comparison of performance with different loss functions is presented. The new loss functions presented in this paper are shown to outperform Cross-entropy loss on computer vision and NLP benchmarks.

翻译：所有机器学习算法均使用损失、成本、效用或奖励函数来编码学习目标并监督学习过程。这一监督学习的函数常被忽视，它决定了错误输出如何被惩罚，并可通过调优提升性能。本文表明，神经网络的训练速度与最终准确率显著依赖于训练时所用的损失函数。具体而言，不同损失函数下的导数数值存在显著差异，导致基于梯度下降的反向传播训练后性能差异显著。本文探讨了相较于流行的交叉熵损失，采用更宽松或更严格的损失函数对性能的影响。提出了八种新损失函数，并展示了不同损失函数下的性能对比。实验表明，本文提出的新损失函数在计算机视觉与自然语言处理基准测试中的表现均优于交叉熵损失。