Due to long generations, large language model (LLM) math reasoning demands significant computational resources and time. While many existing efficient inference methods have been developed with excellent performance preservation on language tasks, they often severely degrade math performance. In this paper, we propose Caprese, a resource-efficient distillation method to recover lost capabilities from deploying efficient inference methods, focused primarily in feedforward blocks. With original weights unperturbed, roughly 1% of additional parameters, and only 20K synthetic training samples, we are able to recover much if not all of the reasoning capabilities lost from efficient inference for thinking LLMs and without harm to language tasks for instruct LLMs. Moreover, Caprese slashes the number of active parameters (~2B cut for Gemma 2 9B and Llama 3.1 8B) and integrates cleanly into existing model layers to reduce latency (>16% time-to-next-token reduction) while encouraging response brevity (up to 8.5% fewer tokens).

翻译：由于生成长度较大，大语言模型（LLM）的数学推理任务需要大量计算资源和时间。尽管现有许多高效推理方法在语言任务上实现了优异的性能保持，但它们往往严重损害数学推理性能。本文提出Caprese，一种资源高效的蒸馏方法，旨在恢复因部署高效推理方法而损失的能力，主要聚焦于前馈模块。该方法不扰动原始权重，仅增加约1%的参数，并使用仅2万条合成训练样本，即可为思维型LLM恢复高效推理所损失的大部分（若非全部）推理能力，且对指令型LLM的语言任务无损害。此外，Caprese大幅削减了激活参数量（Gemma 2 9B和Llama 3.1 8B模型约减少20亿参数），并能无缝集成到现有模型层中以降低延迟（下一词元生成时间减少超过16%），同时促进响应简洁性（生成词元数最多减少8.5%）。