Virtually all machine learning tasks are characterized using some form of loss function, and "good performance" is typically stated in terms of a sufficiently small average loss, taken over the random draw of test data. While optimizing for performance on average is intuitive, convenient to analyze in theory, and easy to implement in practice, such a choice brings about trade-offs. In this work, we survey and introduce a wide variety of non-traditional criteria used to design and evaluate machine learning algorithms, place the classical paradigm within the proper historical context, and propose a view of learning problems which emphasizes the question of "what makes for a desirable loss distribution?" in place of tacit use of the expected loss.

翻译：几乎所有机器学习任务都通过某种损失函数来定义特征，"良好性能"通常表现为对随机抽取的测试数据取平均后足够小的平均损失。尽管以平均性能为优化目标具有直观性、理论分析便利性与实践易操作性，但这种选择也带来了权衡取舍。本文系统综述了用于设计与评估机器学习算法的各类非传统准则，将经典范式置于恰当的历史脉络中，并提出一种强调"何为理想的损失分布"问题的学习任务视角，以替代对期望损失的含蓄使用。