We explore two differentiable deep declarative layers, namely least squares on sphere (LESS) and implicit eigen decomposition (IED), for learning the principal matrix features (PMaF). This can be used to represent data features with a low-dimension vector containing dominant information from a high-dimension matrix. We first solve the problems with iterative optimization in the forward pass and then backpropagate the solution for implicit gradients under a bi-level optimization framework. Particularly, adaptive descent steps with the backtracking line search method and descent decay in the tangent space are studied to improve the forward pass efficiency of LESS. Meanwhile, exploited data structures are used to greatly reduce the computational complexity in the backward pass of LESS and IED. Empirically, we demonstrate the superiority of our layers over the off-the-shelf baselines by comparing the solution optimality and computational requirements.

翻译：我们探索了两种可微分的深度声明式层，即球面最小二乘（LESS）与隐式特征分解（IED），用于学习主矩阵特征（PMaF）。该方法能够以包含高维矩阵主导信息的低维向量表示数据特征。我们首先在前向传播中通过迭代优化求解问题，随后在双层优化框架下利用反向传播求解隐式梯度。特别地，我们研究了基于回溯线搜索的自适应下降步长以及切空间内的下降衰减策略，以提升LESS层的前向传播效率。同时，通过利用数据结构的特性，大幅降低了LESS与IED层后向传播的计算复杂度。实验结果表明，与现有基线方法相比，本方法在解的最优性与计算需求方面均具有显著优势。