Large Language Models (LLMs) are composed of neurons that exhibit various behaviors and roles, which become increasingly diversified as models scale. Recent studies have revealed that not all neurons are active across different datasets, and this sparsity correlates positively with the task-specific ability, leading to advancements in model pruning and training efficiency. Traditional fine-tuning methods engage all parameters of LLMs, which is computationally expensive and may not be necessary. In contrast, Parameter-Efficient Fine-Tuning (PEFT) approaches aim to minimize the number of trainable parameters, yet they still operate at a relatively macro scale (e.g., layer-level). We introduce Neuron-Level Fine-Tuning (NeFT), a novel approach that refines the granularity of parameter training down to the individual neuron, enabling more precise and computationally efficient model updates. The experimental results show that NeFT not only exceeded the performance of full-parameter fine-tuning and PEFT but also provided insights into the analysis of neurons.

翻译：大语言模型由展现出不同行为与角色的神经元构成，且随着模型规模扩大，这些差异愈发多样化。近期研究表明，并非所有神经元在不同数据集上均保持活跃，这种稀疏性与任务特定能力呈正相关，从而推动了模型剪枝与训练效率的提升。传统微调方法需调整大语言模型的所有参数，既耗费计算资源又非必要。参数高效微调方法虽旨在最小化可训练参数数量，但仍处于相对宏观的粒度（如层级层面）。本文提出神经元级微调方法——通过将参数训练粒度细化至单个神经元，实现更精准且计算高效的模型更新。实验结果表明，NeFT不仅超越全参数微调与参数高效微调的性能，更为神经元分析提供了新视角。