We explore two differentiable deep declarative layers, namely least squares on sphere (LESS) and implicit eigen decomposition (IED), for learning the principal matrix features (PMaF). This can be used to represent data features with a low-dimension vector containing dominant information from a high-dimension matrix. We first solve the problems with iterative optimization in the forward pass and then backpropagate the solution for implicit gradients under a bi-level optimization framework. Particularly, adaptive descent steps with the backtracking line search method and descent decay in the tangent space are studied to improve the forward pass efficiency of LESS. Meanwhile, exploited data structures are used to greatly reduce the computational complexity in the backward pass of LESS and IED. Empirically, we demonstrate the superiority of our layers over the off-the-shelf baselines by comparing the solution optimality and computational requirements.

翻译：我们探索了两种可微分的深度声明式层，即球面上的最小二乘（LESS）和隐式特征分解（IED），用于学习主矩阵特征（PMaF）。这可用于通过包含高维矩阵中主导信息的低维向量来表示数据特征。我们首先在前向传播中通过迭代优化解决问题，然后在双层优化框架下反向传播解的隐式梯度。特别地，我们研究了采用回溯线性搜索方法和切空间下降衰减的自适应下降步长，以提高LESS前向传播的效率。同时，利用所发掘的数据结构大大降低了LESS和IED后向传播的计算复杂度。通过实验，我们比较了解的最优性和计算需求，证明了我们的层优于现成的基线方法。