Classification and probability estimation are fundamental tasks with broad applications across modern machine learning and data science, spanning fields such as biology, medicine, engineering, and computer science. Recent development of weighted Support Vector Machines (wSVMs) has demonstrated considerable promise in robustly and accurately predicting class probabilities and performing classification across a variety of problems (Wang et al., 2008). However, the existing framework relies on an $\ell^2$-norm regularized binary wSVMs optimization formulation, which is designed for dense features and exhibits limited performance in the presence of sparse features with redundant noise. Effective sparse learning thus requires prescreening of important variables for each binary wSVM to ensure accurate estimation of pairwise conditional probabilities. In this paper, we propose a novel class of wSVMs frameworks that incorporate automatic variable selection with accurate probability estimation for sparse learning problems. We developed efficient algorithms for variable selection by solving either the $\ell^1$-norm or elastic net regularized wSVMs optimization problems. Class probability is then estimated either via the $\ell^2$-norm regularized wSVMs framework applied to the selected variables, or directly through elastic net regularized wSVMs. The two-step approach offers a strong advantage in simultaneous automatic variable selection and reliable probability estimators with competitive computational efficiency. The elastic net regularized wSVMs achieve superior performance in both variable selection and probability estimation, with the added benefit of variable grouping, at the cost of increases compensation time for high dimensional settings. The proposed wSVMs-based sparse learning methods are broadly applicable and can be naturally extended to $K$-class problems through ensemble learning.

翻译：分类与概率估计是现代机器学习与数据科学中的基础任务，广泛应用于生物学、医学、工程学和计算机科学等领域。近期发展的加权支持向量机（wSVMs）在多种问题中展现出稳健且准确预测类别概率及执行分类的显著潜力（Wang等，2008）。然而，现有框架基于$\ell^2$-范数正则化的二值wSVMs优化公式，该公式针对稠密特征设计，在存在冗余噪声的稀疏特征场景下表现有限。因此，有效的稀疏学习需要对每个二值wSVM进行关键变量的预筛选，以确保成对条件概率的准确估计。本文提出一类新颖的wSVMs框架，融合自动变量选择与精确概率估计以解决稀疏学习问题。我们通过求解$\ell^1$-范数或弹性网络正则化的wSVMs优化问题，开发了高效的变量选择算法。随后，类别概率可通过两种方式估计：对所选变量应用$\ell^2$-范数正则化wSVMs框架，或直接通过弹性网络正则化wSVMs。两阶段方法在同时实现自动变量选择与可靠概率估计方面展现出显著优势，且保持竞争性的计算效率。弹性网络正则化wSVMs在变量选择与概率估计中均表现优异，额外具备变量分组能力，其代价是在高维场景下补偿时间增加。本文提出的基于wSVMs的稀疏学习方法具有广泛适用性，可通过集成学习自然扩展至$K$类问题。