Large Language Models (LLMs) have shown significant promise as copilots in various tasks. Local deployment of LLMs on edge devices is necessary when handling privacy-sensitive data or latency-sensitive tasks. The computational constraints of such devices make direct deployment of powerful large-scale LLMs impractical, necessitating the Knowledge Distillation from large-scale models to lightweight models. Lots of work has been done to elicit diversity and quality training examples from LLMs, but little attention has been paid to aligning teacher instructional content based on student preferences, akin to "responsive teaching" in pedagogy. Thus, we propose ARTE, dubbed Aligning TeacheR with StudenT PreferencEs, a framework that aligns the teacher model with student preferences to generate tailored training examples for Knowledge Distillation. Specifically, we elicit draft questions and rationales from the teacher model, then collect student preferences on these questions and rationales using students' performance with in-context learning as a proxy, and finally align the teacher model with student preferences. In the end, we repeat the first step with the aligned teacher model to elicit tailored training examples for the student model on the target task. Extensive experiments on academic benchmarks demonstrate the superiority of ARTE over existing instruction-tuning datasets distilled from powerful LLMs. Moreover, we thoroughly investigate the generalization of ARTE, including the generalization of fine-tuned student models in reasoning ability and the generalization of aligned teacher models to generate tailored training data across tasks and students. In summary, our contributions lie in proposing a novel framework for tailored training example generation, demonstrating its efficacy in experiments, and investigating the generalization of both student & aligned teacher models in ARTE.

翻译：大型语言模型（LLM）作为协同工具已在多种任务中展现出巨大潜力。在处理隐私敏感数据或延迟敏感任务时，需将LLM本地部署于边缘设备。此类设备的计算限制使得直接部署大规模LLM不可行，因此需要从大规模模型向轻量级模型进行知识蒸馏。现有研究已从LLM中提取出多样化的高质量训练样本，但鲜有工作关注如何根据学生偏好调整教师的教学内容——这类似于教育学中的"响应式教学"。为此，我们提出ARTE框架（基于学生偏好对齐教师模型），通过将教师模型与学生偏好对齐，为知识蒸馏生成定制化训练样本。具体而言，我们首先从教师模型中生成初始问题与推理过程，随后以学生在上下文学习中的表现为代理指标收集其对问题与推理的偏好，最终使教师模型与学生偏好对齐。最后，利用对齐后的教师模型重新执行第一步，为目标任务中的学生模型生成定制化训练样本。在学术基准上的大量实验表明，ARTE优于现有从强大LLM蒸馏得到的指令调优数据集。此外，我们深入研究了ARTE的泛化能力，包括微调后学生模型在推理能力上的泛化性，以及对齐后教师模型跨任务与学生生成定制化训练数据的泛化性。综上所述，我们的贡献在于：提出了一种新颖的定制化训练样本生成框架，通过实验验证其有效性，并深入探究了ARTE中学生模型与对齐教师模型的泛化特性。