Cost-sensitive loss functions are crucial in many real-world prediction problems, where different types of errors are penalized differently; for example, in medical diagnosis, a false negative prediction can lead to worse consequences than a false positive prediction. However, traditional PAC learning theory has mostly focused on the symmetric 0-1 loss, leaving cost-sensitive losses largely unaddressed. In this work, we extend the celebrated theory of boosting to incorporate both cost-sensitive and multi-objective losses. Cost-sensitive losses assign costs to the entries of a confusion matrix, and are used to control the sum of prediction errors accounting for the cost of each error type. Multi-objective losses, on the other hand, simultaneously track multiple cost-sensitive losses, and are useful when the goal is to satisfy several criteria at once (e.g., minimizing false positives while keeping false negatives below a critical threshold). We develop a comprehensive theory of cost-sensitive and multi-objective boosting, providing a taxonomy of weak learning guarantees that distinguishes which guarantees are trivial (i.e., can always be achieved), which ones are boostable (i.e., imply strong learning), and which ones are intermediate, implying non-trivial yet not arbitrarily accurate learning. For binary classification, we establish a dichotomy: a weak learning guarantee is either trivial or boostable. In the multiclass setting, we describe a more intricate landscape of intermediate weak learning guarantees. Our characterization relies on a geometric interpretation of boosting, revealing a surprising equivalence between cost-sensitive and multi-objective losses.

翻译：代价敏感损失函数在许多现实世界预测问题中至关重要，其中不同类型的错误被赋予不同的惩罚权重；例如在医疗诊断中，假阴性预测可能比假阳性预测导致更严重的后果。然而，传统的PAC学习理论主要关注对称的0-1损失，对代价敏感损失的处理尚不充分。本研究将经典的提升理论扩展至同时涵盖代价敏感损失与多目标损失。代价敏感损失为混淆矩阵中的各项分配代价，用于控制各类错误按代价加权后的预测误差总和。另一方面，多目标损失同时追踪多个代价敏感损失，适用于需要同时满足多项准则的场景（例如在将假阴性控制在临界阈值以下的同时最小化假阳性）。我们建立了完整的代价敏感与多目标提升理论体系，提出了一套弱学习保证的分类框架，用以区分哪些保证是平凡的（即总能达成），哪些是可提升的（即能推导出强学习），哪些处于中间状态（即能实现非平凡但非任意精确的学习）。针对二分类问题，我们证明了一个二分定理：弱学习保证要么是平凡的，要么是可提升的。在多分类场景中，我们揭示了更为复杂的中间弱学习保证谱系。该理论表征基于对提升的几何解释，并意外地发现了代价敏感损失与多目标损失之间的等价关系。