Learning Using Privileged Information is a particular type of knowledge distillation where the teacher model benefits from an additional data representation during training, called privileged information, improving the student model, which does not see the extra representation. However, privileged information is rarely available in practice. To this end, we propose a text classification framework that harnesses text-to-image diffusion models to generate artificial privileged information. The generated images and the original text samples are further used to train multimodal teacher models based on state-of-the-art transformer-based architectures. Finally, the knowledge from multimodal teachers is distilled into a text-based (unimodal) student. Hence, by employing a generative model to produce synthetic data as privileged information, we guide the training of the student model. Our framework, called Learning Using Generated Privileged Information (LUGPI), yields noticeable performance gains on four text classification data sets, demonstrating its potential in text classification without any additional cost during inference.

翻译：学习使用特权信息是一种特殊的知识蒸馏方式，其中教师模型在训练过程中得益于一种额外的数据表示（称为特权信息），从而改进了学生模型（该模型未看到这种额外表示）。然而，特权信息在实践中很少可用。为此，我们提出了一种文本分类框架，该框架利用文本到图像扩散模型生成人工特权信息。生成的图像和原始文本样本进一步用于训练基于最先进的Transformer架构的多模态教师模型。最后，多模态教师的知识被蒸馏到基于文本（单模态）的学生模型中。因此，通过使用生成模型生成合成数据作为特权信息，我们指导了学生模型的训练。我们的框架称为“使用生成的特权信息的学习”（LUGPI），在四个文本分类数据集上取得了显著的性能提升，展示了其在文本分类中的潜力，且推理过程中无需任何额外成本。