All machine learning algorithms use a loss, cost, utility or reward function to encode the learning objective and oversee the learning process. This function that supervises learning is a frequently unrecognized hyperparameter that determines how incorrect outputs are penalized and can be tuned to improve performance. This paper shows that training speed and final accuracy of neural networks can significantly depend on the loss function used to train neural networks. In particular derivative values can be significantly different with different loss functions leading to significantly different performance after gradient descent based Backpropagation (BP) training. This paper explores the effect on performance of using new loss functions that are also convex but penalize errors differently compared to the popular Cross-entropy loss. Two new classification loss functions that significantly improve performance on a wide variety of benchmark tasks are proposed. A new loss function call smooth absolute error that outperforms the Squared error, Huber and Log-Cosh losses on datasets with significantly many outliers is proposed. This smooth absolute error loss function is infinitely differentiable and more closely approximates the absolute error loss compared to the Huber and Log-Cosh losses used for robust regression.

翻译：所有机器学习算法均使用损失函数、代价函数、效用函数或奖励函数来编码学习目标并监督学习过程。监督学习的这一函数常被忽视，实则为决定错误输出惩罚方式的可调超参数，通过调整可提升模型性能。本文表明，神经网络训练速度与最终精度显著依赖于训练所用的损失函数。特别地，不同损失函数下的导数值差异显著，导致基于梯度下降的反向传播（BP）训练后性能存在显著差异。本文探究了采用同样凸性但惩罚误差方式与常用交叉熵损失不同的新型损失函数对性能的影响。提出了两种新的分类损失函数，在多种基准任务上显著提升了性能。针对存在大量异常值的数据集，提出了一种称为“平滑绝对误差”的新损失函数，其性能优于平方误差、Huber损失和Log-Cosh损失。该平滑绝对误差损失函数具有无限可微性，且相比用于鲁棒回归的Huber损失和Log-Cosh损失，能更精确地逼近绝对误差损失。