Nonlinear matrix decomposition (NMD) with the ReLU function, denoted ReLU-NMD, is the following problem: given a sparse, nonnegative matrix $X$ and a factorization rank $r$, identify a rank-$r$ matrix $\Theta$ such that $X\approx \max(0,\Theta)$. This decomposition finds application in data compression, matrix completion with entries missing not at random, and manifold learning. The standard ReLU-NMD model minimizes the least squares error, that is, $\|X - \max(0,\Theta)\|_F^2$. The corresponding optimization problem is nondifferentiable and highly nonconvex. This motivated Saul to propose an alternative model, Latent-ReLU-NMD, where a latent variable $Z$ is introduced and satisfies $\max(0,Z)=X$ while minimizing $\|Z - \Theta\|_F^2$ (``A nonlinear matrix decomposition for mining the zeros of sparse data'', SIAM J. Math. Data Sci., 2022). Our first contribution is to show that the two formulations may yield different low-rank solutions $\Theta$; in particular, we show that Latent-ReLU-NMD can be ill-posed when ReLU-NMD is not, meaning that there are instances in which the infimum of Latent-ReLU-NMD is not attained while that of ReLU-NMD is. We also consider another alternative model, called 3B-ReLU-NMD, which parameterizes $\Theta=WH$, where $W$ has $r$ columns and $H$ has $r$ rows, allowing one to get rid of the rank constraint in Latent-ReLU-NMD. Our second contribution is to prove the convergence of a block coordinate descent (BCD) applied to 3B-ReLU-NMD and referred to as BCD-NMD. Our third contribution is a novel extrapolated variant of BCD-NMD, dubbed eBCD-NMD, which we prove is also convergent under mild assumptions. We illustrate the significant acceleration effect of eBCD-NMD compared to BCD-NMD, and also show that eBCD-NMD performs well against the state of the art on synthetic and real-world data sets.

翻译：采用ReLU函数的非线性矩阵分解（记为ReLU-NMD）可表述为：给定稀疏非负矩阵$X$与分解秩$r$，寻找秩为$r$的矩阵$\Theta$使得$X\approx \max(0,\Theta)$。该分解在数据压缩、非随机缺失项的矩阵补全以及流形学习中具有应用价值。标准ReLU-NMD模型最小化最小二乘误差，即$\|X - \max(0,\Theta)\|_F^2$。对应的优化问题不可微且高度非凸。为此，Saul提出了一种替代模型——隐变量ReLU-NMD（Latent-ReLU-NMD），通过引入满足$\max(0,Z)=X$的隐变量$Z$，并最小化$\|Z - \Theta\|_F^2$（参见《用于稀疏数据零值挖掘的非线性矩阵分解》，SIAM数据科学数学杂志，2022年）。我们的第一个贡献在于证明两种模型可能产生不同的低秩解$\Theta$；特别地，我们证明了当ReLU-NMD适定时，Latent-ReLU-NMD可能不适定，即存在Latent-ReLU-NMD的下确界不可达到而ReLU-NMD可达到的实例。我们还研究了另一种称为3B-ReLU-NMD的替代模型，该模型将$\Theta$参数化为$WH$，其中$W$具有$r$列、$H$具有$r$行，从而消除了Latent-ReLU-NMD中的秩约束。我们的第二个贡献是证明了应用于3B-ReLU-NMD的块坐标下降法（BCD），即BCD-NMD的收敛性。第三个贡献是提出了一种新的外推式BCD-NMD变体（eBCD-NMD），并在温和假设下证明了其收敛性。我们通过实验展示了eBCD-NMD相比BCD-NMD的显著加速效果，并证明eBCD-NMD在合成与真实数据集上均优于现有先进方法。