Model pruning is an effective approach for compressing large language models (LLMs). However, this process often leads to significant degradation of model capabilities. While post-training techniques such as instruction tuning are commonly employed to recover model performance, existing methods often overlook the uneven deterioration of model capabilities and incur high computational costs. Moreover, some irrelevant instructions may also introduce negative effects to model capacity recovery. To address these challenges, we propose the \textbf{P}ost-training d\textbf{A}ta \textbf{S}election method for \textbf{E}fficient pruned large language model \textbf{R}ecovery (\textbf{PASER}). PASER aims to identify instructions to recover the most compromised model capacities with a certain data budget. Our approach first applies manifold learning and spectral clustering to group recovery instructions in the semantic space, revealing capability-specific instruction sets. Then, the data budget is adaptively allocated across clusters by the degree of corresponding model capability degradation. In each cluster, we prioritize data samples that lead to the most decline of model performance. To mitigate potential negative tuning effects, we also detect and filter out conflicting or irrelevant recovery data. Extensive experiments demonstrate that PASER significantly outperforms conventional baselines, effectively recovering the general capabilities of pruned LLMs while utilizing merely 4\%-20\% of the original post-training data. We provide the code repository in \href{https://github.com/BokwaiHo/PASER}{Link}.

翻译：模型剪枝是压缩大语言模型（LLMs）的有效方法。然而，该过程通常会导致模型能力显著退化。尽管指令微调等后训练技术常被用于恢复模型性能，但现有方法往往忽视模型能力的不均衡衰退，且计算成本高昂。此外，部分无关指令也可能对模型能力恢复产生负面影响。为应对这些挑战，我们提出面向高效剪枝大语言模型恢复的后训练数据选择方法（PASER）。PASER旨在通过有限数据预算识别能够恢复受损最严重模型能力的指令。我们的方法首先应用流形学习与谱聚类在语义空间中对恢复指令进行分组，从而揭示面向特定能力的指令集合。随后，数据预算根据对应模型能力的衰退程度在聚类间进行自适应分配。在每个聚类中，我们优先选择导致模型性能下降最显著的数据样本。为缓解潜在的负向调优效应，我们还检测并过滤冲突或无关的恢复数据。大量实验表明，PASER显著优于传统基线方法，仅需使用原始后训练数据的4%-20%即可有效恢复剪枝后LLMs的通用能力。代码仓库发布于\href{https://github.com/BokwaiHo/PASER}{链接}。