In view of the huge number of parameters of Large language models (LLMs) , tuning all parameters is very costly, and accordingly fine-tuning specific parameters is more sensible. Most of parameter efficient fine-tuning (PEFT) concentrate on parameter selection strategies, such as additive method, selective method and reparametrization-based method. However, there are few methods that consider the impact of data samples on parameter selecting, such as Fish Mask based method. Fish Mask randomly choose a part of data samples and treat them equally during parameter selection, which is unable to dynamically select optimal parameters for inconstant data distributions. In this work, we adopt a data-oriented perspective, then proposing an IRD ($\mathrm{\underline I}$terative sample-parameter $\mathrm{\underline R}$ange $\mathrm{\underline D}$ecreasing) algorithm to search the best setting of sample-parameter pair for FISH Mask. In each iteration, by searching the set of samples and parameters with larger Fish information, IRD can find better sample-parameter pair in most scale. We demonstrate the effectiveness and rationality of proposed strategy by conducting experiments on GLUE benchmark. Experimental results show our strategy optimizes the parameter selection and achieves preferable performance.

翻译：鉴于大语言模型（LLMs）参数规模巨大，全参数微调成本高昂，因此针对特定参数进行微调更为合理。大多数参数高效微调（PEFT）方法集中于参数选择策略，例如加法方法、选择方法和基于重参数化的方法。然而，鲜有方法考虑数据样本对参数选择的影响，例如基于FISH掩码的方法。FISH掩码在参数选择过程中随机选取部分数据样本并同等对待，无法针对动态变化的数据分布动态选择最优参数。在本工作中，我们采用面向数据的视角，提出一种IRD（$\mathrm{\underline I}$迭代样本-$\mathrm{\underline R}$参数范围$\mathrm{\underline D}$递减）算法，以搜索FISH掩码的最优样本-参数对配置。在每次迭代中，通过搜索具有更大Fish信息的样本集和参数集，IRD能够在大多数尺度下找到更优的样本-参数对。通过在GLUE基准上的实验，我们验证了所提策略的有效性和合理性。实验结果表明，本文策略优化了参数选择过程，并取得了更优的性能。