Adapting large-scale pre-trained generative models in a parameter-efficient manner is gaining traction. Traditional methods like low rank adaptation achieve parameter efficiency by imposing constraints but may not be optimal for tasks requiring high representation capacity. We propose a novel spectrum-aware adaptation framework for generative models. Our method adjusts both singular values and their basis vectors of pretrained weights. Using the Kronecker product and efficient Stiefel optimizers, we achieve parameter-efficient adaptation of orthogonal matrices. We introduce Spectral Orthogonal Decomposition Adaptation (SODA), which balances computational efficiency and representation capacity. Extensive evaluations on text-to-image diffusion models demonstrate SODA's effectiveness, offering a spectrum-aware alternative to existing fine-tuning methods.

翻译：以参数高效的方式适配大规模预训练生成模型正获得广泛关注。传统方法如低秩适应通过施加约束实现参数高效性，但对于需要高表示容量的任务可能并非最优。我们提出了一种新颖的频谱感知适配框架用于生成模型。我们的方法同时调整预训练权重的奇异值及其基向量。利用克罗内克积和高效的Stiefel优化器，我们实现了正交矩阵的参数高效适配。我们引入了频谱正交分解适配（SODA），该方法在计算效率与表示容量之间取得平衡。在文本到图像扩散模型上的大量评估证明了SODA的有效性，为现有微调方法提供了一种频谱感知的替代方案。