Adapting pre-trained foundation models for various downstream tasks has been prevalent in artificial intelligence. Due to the vast number of tasks and high costs, adjusting all parameters becomes unfeasible. To mitigate this, several fine-tuning techniques have been developed to update the pre-trained model weights in a more resource-efficient manner, such as through low-rank adjustments. Yet, almost all of these methods focus on linear weights, neglecting the intricacies of parameter spaces in higher dimensions like 4D. Alternatively, some methods can be adapted for high-dimensional parameter space by compressing changes in the original space into two dimensions and then employing low-rank matrix decomposition. However, these approaches destructs the structural integrity of the involved high-dimensional spaces. To tackle the diversity of dimensional spaces across different foundation models and provide a more precise representation of the changes within these spaces, this paper introduces a generalized parameter-efficient fine-tuning framework, FLoRA, designed for various dimensional parameter space. Specifically, utilizing Tucker decomposition, FLoRA asserts that changes in each dimensional parameter space are based on a low-rank core space which maintains the consistent topological structure with the original space. It then models the changes through this core space alongside corresponding weights to reconstruct alterations in the original space. FLoRA effectively preserves the structural integrity of the change of original N-dimensional parameter space, meanwhile decomposes it via low-rank tensor decomposition. Extensive experiments on computer vision, natural language processing and multi-modal tasks validate FLoRA's effectiveness. Codes are available at https://github.com/SJTU-DeepVisionLab/FLoRA.

翻译：在人工智能领域，将预训练的基础模型适配到各种下游任务已十分普遍。由于任务数量庞大且成本高昂，调整所有参数变得不可行。为缓解此问题，已开发出多种微调技术，以更高效的方式更新预训练模型权重，例如通过低秩调整。然而，这些方法几乎都聚焦于线性权重，忽略了更高维度（如4D）参数空间的复杂性。另一些方法则可通过将原始空间的变化压缩至二维，再采用低秩矩阵分解来适配高维参数空间，但这些方法破坏了所涉高维空间的结构完整性。为应对不同基础模型中维度空间的多样性，并更精确地表征这些空间内的变化，本文提出了一种通用的参数高效微调框架FLoRA，专为各种维度的参数空间设计。具体而言，FLoRA利用Tucker分解，主张每个维度参数空间的变化都基于一个与原始空间保持拓扑结构一致的低秩核心空间，并通过该核心空间及相应权重来建模变化，以重构原始空间中的改变。FLoRA有效保持了原始N维参数空间变化的结构完整性，同时通过低秩张量分解对其进行分解。在计算机视觉、自然语言处理和多模态任务上的大量实验验证了FLoRA的有效性。代码可在https://github.com/SJTU-DeepVisionLab/FLoRA获取。