Consider an active learning setting in which a learner has a training set with few labeled examples and a pool set with many unlabeled inputs, while a remote teacher has a pre-trained model that is known to perform well for the learner's task. The learner actively transmits batches of unlabeled inputs to the teacher through a constrained communication channel for labeling. This paper addresses the following key questions: (i) Active batch selection: Which batch of inputs should be sent to the teacher to acquire the most useful information and thus reduce the number of required communication rounds? (ii) Batch encoding: How do we encode the batch of inputs for transmission to the teacher to reduce the communication resources required at each round? We introduce Communication-Constrained Bayesian Active Knowledge Distillation (CC-BAKD), a novel protocol that integrates Bayesian active learning with compression via a linear mix-up mechanism. Bayesian active learning selects the batch of inputs based on their epistemic uncertainty, addressing the "confirmation bias" that is known to increase the number of required communication rounds. Furthermore, the proposed mix-up compression strategy is integrated with the epistemic uncertainty-based active batch selection process to reduce the communication overhead per communication round.

翻译：考虑一个主动学习场景，其中学习者拥有少量标注样本的训练集和大量未标注输入的池集，而远程教师拥有一个已知在该学习者任务上表现良好的预训练模型。学习者通过受限通信信道主动向教师传输未标注输入的批次进行标注。本文解决以下关键问题：（i）主动批次选择：应选择哪批输入发送给教师以获取最有用的信息，从而减少所需通信轮数？（ii）批次编码：如何对传输给教师的输入批次进行编码，以降低每轮通信所需的资源？我们提出通信受限贝叶斯主动知识蒸馏（CC-BAKD），一种结合贝叶斯主动学习与基于线性混合机制的压缩的新型协议。贝叶斯主动学习根据输入的认知不确定性选择批次，解决了已知会增加所需通信轮数的“确认偏差”问题。此外，所提出的混合压缩策略与基于认知不确定性的主动批次选择过程相集成，以降低每轮通信的开销。