Universal image representations are critical in enabling real-world fine-grained and instance-level recognition applications, where objects and entities from any domain must be identified at large scale. Despite recent advances, existing methods fail to capture important domain-specific knowledge, while also ignoring differences in data distribution across different domains. This leads to a large performance gap between efficient universal solutions and expensive approaches utilising a collection of specialist models, one for each domain. In this work, we make significant strides towards closing this gap, by introducing a new learning technique, dubbed UDON (Universal Dynamic Online DistillatioN). UDON employs multi-teacher distillation, where each teacher is specialized in one domain, to transfer detailed domain-specific knowledge into the student universal embedding. UDON's distillation approach is not only effective, but also very efficient, by sharing most model parameters between the student and all teachers, where all models are jointly trained in an online manner. UDON also comprises a sampling technique which adapts the training process to dynamically allocate batches to domains which are learned slower and require more frequent processing. This boosts significantly the learning of complex domains which are characterised by a large number of classes and long-tail distributions. With comprehensive experiments, we validate each component of UDON, and showcase significant improvements over the state of the art in the recent UnED benchmark. Code: https://github.com/nikosips/UDON .

翻译：摘要：通用图像表示对于实现现实世界中的细粒度及实例级识别应用至关重要，此类应用需在大规模范围内识别任意领域的对象与实体。尽管近期研究取得了进展，现有方法仍无法捕获关键的领域特定知识，同时忽略了不同领域间数据分布的差异。这导致高效通用方案与使用多专家模型集合（每个领域对应一个专用模型）的高成本方法之间存在巨大的性能差距。在本工作中，我们通过引入一种名为UDON（通用动态在线蒸馏）的新型学习技术，向缩小这一差距迈出了重要一步。UDON采用多教师蒸馏策略——每位教师专精于一个领域——将细粒度领域特定知识迁移至学生通用嵌入模型中。通过让学生与所有教师共享大部分模型参数，并以在线方式联合训练所有模型，UDON的蒸馏方法不仅高效，且极具效率。UDON还包含一种采样技术，可自适应调整训练过程，动态将批次分配给学习速度较慢、需更频繁处理的领域。这显著提升了复杂领域（以类别数量庞大和长尾分布为特征）的学习效果。通过全面实验，我们验证了UDON各组成部分的有效性，并展示了其在近期UnED基准测试中相较于现有技术的显著改进。代码地址：https://github.com/nikosips/UDON。