Fine-tuning all parameters of Large Language Models (LLMs) is computationally expensive. Parameter-Efficient Fine-Tuning (PEFT) methods address this by selectively fine-tuning specific parameters. Most of the parameter efficient fine-tuning (PEFT) methods center on selecting or introducing a set of parameters to be fine-tuned. However, there are few methods that consider the impact of data samples on parameter selecting. Representative data driven methods include FISH Mask based method, which randomly selects a portion of data samples as a basis when selecting parameters. However, this random data sample selection method cannot select optimal parameters for unstable data distribution. In this work, we introduce a data-centric approach and propose the Iterative Range Decreasing (IRD) algorithm to optimize the sample-parameter pair selection in FISH Mask. IRD iteratively refines the selection by identifying subsets of samples and parameters exhibiting higher Fisher information. We demonstrate the effectiveness and rationality of proposed strategy by conducting experiments on GLUE benchmark. Experimental results show our strategy optimizes the parameter selection and achieves preferable performance over some typical baseline methods.

翻译：大型语言模型（LLM）的全参数微调计算成本高昂。参数高效微调（PEFT）方法通过选择性地微调特定参数来解决这一问题。大多数参数高效微调（PEFT）方法侧重于选择或引入一组待微调的参数。然而，少有方法考虑数据样本对参数选择的影响。代表性的数据驱动方法包括基于FISH Mask的方法，其在选择参数时随机选取一部分数据样本作为依据。然而，这种随机数据样本选择方法无法为不稳定的数据分布选择最优参数。在本工作中，我们引入一种以数据为中心的方法，并提出迭代范围递减（IRD）算法，以优化FISH Mask中的样本-参数对选择。IRD通过识别具有更高Fisher信息的样本和参数子集，迭代地优化选择过程。我们在GLUE基准上进行了实验，证明了所提策略的有效性和合理性。实验结果表明，我们的策略优化了参数选择，并取得了优于一些典型基线方法的性能。