Parameter-Efficient Fine-Tuning (PEFT) methods, especially LoRA, are widely used for adapting pre-trained models to downstream tasks due to their computational and storage efficiency. However, in the context of LoRA and its variants, the potential of activation subspaces corresponding to tail eigenvectors remains substantially under-exploited, which may lead to suboptimal fine-tuning performance. In this work, we propose Astra (Activation-Space Tail-Eigenvector Low-Rank Adaptation), a novel PEFT method that leverages the tail eigenvectors of the model output activations-estimated from a small task-specific calibration set-to construct task-adaptive low-rank adapters. By constraining updates to the subspace spanned by these tail eigenvectors, Astra achieves faster convergence and improved downstream performance with a significantly reduced parameter budget. Extensive experiments across natural language understanding (NLU) and natural language generation (NLG) tasks demonstrate that Astra consistently outperforms existing PEFT baselines across 16 benchmarks and even surpasses full fine-tuning (FFT) in certain scenarios.

翻译：参数高效微调方法，特别是LoRA，因其计算和存储效率而被广泛用于使预训练模型适应下游任务。然而，在LoRA及其变体的背景下，与尾部特征向量对应的激活子空间的潜力在很大程度上仍未得到充分探索，这可能导致次优的微调性能。在本工作中，我们提出了Astra（激活空间尾部特征向量低秩适配），这是一种新颖的PEFT方法，它利用模型输出激活的尾部特征向量——通过一个小的任务特定校准集估计得到——来构建任务自适应的低秩适配器。通过将更新约束在这些尾部特征向量所张成的子空间内，Astra以显著减少的参数预算实现了更快的收敛和提升的下游性能。在自然语言理解和自然语言生成任务上的大量实验表明，Astra在16个基准测试中始终优于现有的PEFT基线，甚至在特定场景下超越了全参数微调。