We introduce CRAFT (Cross-layer Rank Adaptation via Frozen Tucker), a parameter-efficient fine-tuning (PEFT) method that applies Tucker tensor decomposition to pre-trained attention weight matrices stacked across transformer layers and trains only small square adaptation matrices on the resulting frozen Tucker factors. Existing tensor-based PEFT methods decompose gradient updates: LoTR applies Tucker decomposition with shared factor matrices, while SuperLoRA groups and reshapes $ΔW$ across layers before applying Tucker decomposition. Separately, methods like PiSSA apply SVD to pre-trained weights but operate independently per layer. CRAFT bridges these two lines of work: it performs full Tucker decomposition via Higher-Order SVD (HOSVD) directly on pre-trained weights organized as cross-layer 3D tensors, freezes all resulting factors, and adapts the model through lightweight trainable transformations applied to each factor matrix. Experiments on the GLUE benchmark using RoBERTa-base and RoBERTa-large demonstrate that CRAFT achieves competitive performance with existing methods while requiring only 41K Tucker adaptation parameters--a count independent of model dimension and depth at fixed Tucker ranks.

翻译：本文介绍CRAFT（通过冻结Tucker分解实现跨层秩自适应），一种参数高效微调方法。该方法将Tucker张量分解应用于跨Transformer层堆叠的预训练注意力权重矩阵，并仅在得到的冻结Tucker因子上训练小型方形自适应矩阵。现有基于张量的PEFT方法主要分解梯度更新：LoTR采用共享因子矩阵进行Tucker分解，而SuperLoRA则在应用Tucker分解前对跨层的$ΔW$进行分组和重塑。另一方面，诸如PiSSA等方法对预训练权重应用奇异值分解，但每层独立操作。CRAFT融合了这两类思路：它通过高阶奇异值分解直接对组织为跨层三维张量的预训练权重执行完整Tucker分解，冻结所有生成的因子，并通过应用于每个因子矩阵的轻量级可训练变换来适配模型。在GLUE基准测试中使用RoBERTa-base和RoBERTa-large的实验表明，CRAFT在仅需41K个Tucker自适应参数（该数量在固定Tucker秩下与模型维度和深度无关）的情况下，取得了与现有方法相竞争的性能。