In applications with significant class imbalance or asymmetric costs, metrics such as the $F_β$-measure, AM measure, Jaccard similarity coefficient, and weighted accuracy offer more suitable evaluation criteria than standard binary classification loss. However, optimizing these metrics present significant computational and statistical challenges. Existing approaches often rely on the characterization of the Bayes-optimal classifier, and use threshold-based methods that first estimate class probabilities and then seek an optimal threshold. This leads to algorithms that are not tailored to restricted hypothesis sets and lack finite-sample performance guarantees. In this work, we introduce principled algorithms for optimizing generalized metrics, supported by $H$-consistency and finite-sample generalization bounds. Our approach reformulates metric optimization as a generalized cost-sensitive learning problem, enabling the design of novel surrogate loss functions with provable $H$-consistency guarantees. Leveraging this framework, we develop new algorithms, METRO (Metric Optimization), with strong theoretical performance guarantees. We report the results of experiments demonstrating the effectiveness of our methods compared to prior baselines.

翻译：在类别不平衡显著或成本不对称的应用场景中，诸如$F_β$值、AM度量、Jaccard相似系数以及加权准确率等度量标准，相比标准的二元分类损失函数，提供了更为合适的评估准则。然而，优化这些度量指标带来了显著的计算与统计挑战。现有方法通常依赖于贝叶斯最优分类器的刻画，并采用基于阈值的方法：首先估计类别概率，然后寻找最优阈值。这导致算法未能针对受限假设集进行定制，且缺乏有限样本性能保证。在本研究中，我们提出了优化广义度量的原理性算法，并辅以$H$一致性与有限样本泛化界作为理论支撑。我们的方法将度量优化问题重构为一个广义的成本敏感学习问题，从而能够设计具有可证明$H$一致性保证的新型代理损失函数。基于此框架，我们开发了具有强大理论性能保证的新算法METRO（度量优化）。我们报告了实验结果，证明了我们的方法相较于先前基线方法的有效性。