Large Language Models (LLMs) have demonstrated impressive zero-shot capabilities and versatility in NLP tasks, however they sometimes fail to maintain crucial invariances for specific tasks. One example is permutation sensitivity, where LLMs' outputs may significantly vary depending on the order of the input options. While debiasing techniques can mitigate these issues, and yield better performance and reliability, they often come with a high computational cost at inference. This paper addresses this inefficiency at inference time. The aim is to distill the capabilities of a computationally intensive, debiased, teacher model into a more compact student model. We explore two variants of student models: one based on pure distillation, and the other on an error-correction approach for more complex tasks, where the student corrects a single biased decision from the teacher to achieve a debiased output. Our approach is general and can be applied to both black-box and white-box LLMs. Furthermore, we demonstrate that our compact, encoder-only student models can outperform their larger, biased teacher counterparts, achieving better results with significantly fewer parameters.

翻译：大型语言模型（LLMs）在自然语言处理任务中展现出卓越的零样本能力和通用性，但在特定任务中往往难以维持关键的不变性。其中一个典型例子是排列敏感性——当输入选项顺序改变时，LLMs的输出可能发生显著差异。虽然去偏技术可以缓解这些问题并提升性能与可靠性，但这些方法通常会在推理阶段带来高昂的计算成本。本文针对推理阶段的效率问题展开研究，旨在将计算密集型、经过去偏处理的教师模型的能力蒸馏至更紧凑的学生模型中。我们探索了两种学生模型变体：一种基于纯蒸馏方法，另一种针对复杂任务采用纠错机制——学生通过纠正教师模型中的单一有偏决策来获得去偏输出。该方法具有通用性，可同时适用于黑盒与白盒LLMs。此外，我们证明紧凑的编码器型学生模型能够超越其规模更大、存在偏见的教师模型，在显著减少参数量的同时取得更优性能。