This paper investigates new families of compositional optimization problems, called $\underline{\bf n}$on-$\underline{\bf s}$mooth $\underline{\bf w}$eakly-$\underline{\bf c}$onvex $\underline{\bf f}$inite-sum $\underline{\bf c}$oupled $\underline{\bf c}$ompositional $\underline{\bf o}$ptimization (NSWC FCCO). There has been a growing interest in FCCO due to its wide-ranging applications in machine learning and AI, as well as its ability to address the shortcomings of stochastic algorithms based on empirical risk minimization. However, current research on FCCO presumes that both the inner and outer functions are smooth, limiting their potential to tackle a more diverse set of problems. Our research expands on this area by examining non-smooth weakly-convex FCCO, where the outer function is weakly convex and non-decreasing, and the inner function is weakly-convex. We analyze a single-loop algorithm and establish its complexity for finding an $\epsilon$-stationary point of the Moreau envelop of the objective function. Additionally, we also extend the algorithm to solving novel non-smooth weakly-convex tri-level finite-sum coupled compositional optimization problems, which feature a nested arrangement of three functions. Lastly, we explore the applications of our algorithms in deep learning for two-way partial AUC maximization and multi-instance two-way partial AUC maximization, using empirical studies to showcase the effectiveness of the proposed algorithms.

翻译：本文研究一类新的复合优化问题，称为非光滑弱凸有限和耦合复合优化（NSWC FCCO）。由于FCCO在机器学习和人工智能领域的广泛应用及其弥补基于经验风险最小化的随机算法缺陷的能力，对其研究兴趣日益增长。然而，当前FCCO研究通常假设内层函数与外层函数均为光滑函数，这限制了其处理更广泛问题的潜力。本研究通过考察非光滑弱凸FCCO拓展了这一领域，其中外层函数为弱凸且非递减，内层函数为弱凸。我们分析了一种单循环算法，并建立了其寻找目标函数莫罗包络的ε-平稳点的计算复杂度。此外，我们还将该算法扩展至求解新型的非光滑弱凸三层有限和耦合复合优化问题，该问题具有三层函数的嵌套结构。最后，我们探讨了所提算法在深度学习中的应用，包括双向部分AUC最大化和多示例双向部分AUC最大化问题，并通过实证研究展示了所提算法的有效性。