Language models pre-trained on large self-supervised corpora, followed by task-specific fine-tuning has become the dominant paradigm in NLP. These pre-training datasets often have a one-to-many structure--e.g. in dialogue there are many valid responses for a given context. However, only some of these responses will be desirable in our downstream task. This raises the question of how we should train the model such that it can emulate the desirable behaviours, but not the undesirable ones. Current approaches train in a one-to-one setup--only a single target response is given for a single dialogue context--leading to models only learning to predict the average response, while ignoring the full range of possible responses. Using text-based games as a testbed, our approach, PASA, uses discrete latent variables to capture the range of different behaviours represented in our larger pre-training dataset. We then use knowledge distillation to distil the posterior probability distribution into a student model. This probability distribution is far richer than learning from only the hard targets of the dataset, and thus allows the student model to benefit from the richer range of actions the teacher model has learned. Results show up to 49% empirical improvement over the previous state-of-the-art model on the Jericho Walkthroughs dataset.

翻译：在大型自监督语料上预训练语言模型，随后针对特定任务进行微调，已成为自然语言处理领域的主流范式。这些预训练数据集通常具有一对多结构——例如，在对话中，给定一个上下文可能对应多种有效回复。然而，在我们的下游任务中，只有部分回复是理想的。这引出了一个问题：我们应如何训练模型，使其能够模仿理想行为，同时避免不良行为。当前方法采用一对一设置——仅对单个对话上下文提供单一目标回复——导致模型仅能学习预测平均回复，而忽略了可能的回复全范围。以基于文本的游戏为试验平台，我们的方法PASA利用离散潜变量来捕获更大规模预训练数据集中所呈现的不同行为范围。随后，我们使用知识蒸馏将后验概率分布蒸馏到学生模型中。该概率分布远比仅从数据集的硬目标学习更为丰富，因此使学生模型能够从教师模型学到的更广泛动作范围中受益。结果显示，在Jericho Walkthroughs数据集上，相对于先前最先进的模型，实证改进幅度高达49%。