Classification and probability estimation have broad applications in modern machine learning and data science applications, including biology, medicine, engineering, and computer science. The recent development of a class of weighted Support Vector Machines (wSVMs) has shown great values in robustly predicting the class probability and classification for various problems with high accuracy. The current framework is based on the $\ell^2$-norm regularized binary wSVMs optimization problem, which only works with dense features and has poor performance at sparse features with redundant noise in most real applications. The sparse learning process requires a prescreen of the important variables for each binary wSVMs for accurately estimating pairwise conditional probability. In this paper, we proposed novel wSVMs frameworks that incorporate automatic variable selection with accurate probability estimation for sparse learning problems. We developed efficient algorithms for effective variable selection for solving either the $\ell^1$-norm or elastic net regularized binary wSVMs optimization problems. The binary class probability is then estimated either by the $\ell^2$-norm regularized wSVMs framework with selected variables or by elastic net regularized wSVMs directly. The two-step approach of $\ell^1$-norm followed by $\ell^2$-norm wSVMs show a great advantage in both automatic variable selection and reliable probability estimators with the most efficient time. The elastic net regularized wSVMs offer the best performance in terms of variable selection and probability estimation with the additional advantage of variable grouping in the compensation of more computation time for high dimensional problems. The proposed wSVMs-based sparse learning methods have wide applications and can be further extended to $K$-class problems through ensemble learning.

翻译：分类与概率估计在现代机器学习与数据科学应用中具有广泛用途，涵盖生物学、医学、工程学及计算机科学等领域。近期发展的加权支持向量机（wSVMs）在稳健预测类概率与高精度分类方面展现出重要价值。现有框架基于ℓ²-范数正则化的二分类wSVMs优化问题，但其仅适用于稠密特征，在实际应用中处理含有冗余噪声的稀疏特征时表现欠佳。稀疏学习过程需要为每个二分类wSVMs预先筛选重要变量，以精确估计成对条件概率。本文提出了新型wSVMs框架，将自动变量选择与精确概率估计相结合，用于解决稀疏学习问题。我们开发了高效算法，通过求解ℓ¹-范数或弹性网络正则化的二分类wSVMs优化问题实现有效变量选择。随后，可通过两种方式估计二分类类概率：采用基于选定变量的ℓ²-范数正则化wSVMs框架，或直接通过弹性网络正则化wSVMs。先ℓ¹-范数后ℓ²-范数的两步wSVMs方法在自动变量选择与可靠概率估计方面展现出显著优势，且计算效率最高。弹性网络正则化wSVMs在变量选择与概率估计方面表现最优，其变量分组特性在高维问题中以额外计算时间为代价提供了补偿。本文提出的基于wSVMs的稀疏学习方法具有广泛适用性，可通过集成学习进一步扩展至K类问题。