Current state-of-the-art self-supervised approaches, are effective when trained on individual domains but show limited generalization on unseen domains. We observe that these models poorly generalize even when trained on a mixture of domains, making them unsuitable to be deployed under diverse real-world setups. We therefore propose a general-purpose, lightweight Domain Disentanglement Module (DDM) that can be plugged into any self-supervised encoder to effectively perform representation learning on multiple, diverse domains with or without shared classes. During pre-training according to a self-supervised loss, DDM enforces a disentanglement in the representation space by splitting it into a domain-variant and a domain-invariant portion. When domain labels are not available, DDM uses a robust clustering approach to discover pseudo-domains. We show that pre-training with DDM can show up to 3.5% improvement in linear probing accuracy on state-of-the-art self-supervised models including SimCLR, MoCo, BYOL, DINO, SimSiam and Barlow Twins on multi-domain benchmarks including PACS, DomainNet and WILDS. Models trained with DDM show significantly improved generalization (7.4%) to unseen domains compared to baselines. Therefore, DDM can efficiently adapt self-supervised encoders to provide high-quality, generalizable representations for diverse multi-domain data.

翻译：当前最先进的自监督方法在单一领域训练时表现有效，但面对未见领域时泛化能力有限。我们观察到即便在混合领域上训练，这些模型泛化效果仍然欠佳，难以部署于多样化的现实场景。为此，我们提出一种轻量级通用领域解耦模块（DDM），可嵌入任意自监督编码器，有效对多领域（含共享类别或无共享类别）数据实施表征学习。在依据自监督损失进行预训练时，DDM通过将表征空间划分为域变异部分和域不变部分来实现表征解耦。当领域标签不可用时，DDM采用鲁棒聚类方法发现伪领域。实验证明，在包含PACS、DomainNet和WILDS的多领域基准测试中，集成DDM的SimCLR、MoCo、BYOL、DINO、SimSiam及Barlow Twins等自监督模型，线性探测准确率最高可提升3.5%。与基线模型相比，DDM训练的模型对未见领域具有显著增强的泛化能力（提升7.4%）。因此，DDM能够高效适配自监督编码器，为多样化多领域数据生成高质量、可泛化的表征。