Collecting a multimodal dataset with two paired modalities A and B or B and C is difficult in practice. Obtaining a dataset with three aligned modalities A, B, and C is even more challenging. For example, some public medical datasets have only genetic sequences and microscopic images for one patient, and only genetic sequences and radiological images for another - but no dataset includes both microscopic and radiological images for the same patient. This makes it difficult to integrate and combine all modalities into a large pre-trained neural network. We introduce LoReTTa (Linking mOdalities with a tRansitive and commutativE pre-Training sTrAtegy) to address this understudied problem. Our self-supervised framework combines causal masked modeling with the rules of commutativity and transitivity to transition within and between different modalities. Thus, it can model the relation A -> C with A -> B -> C. Given a dataset containing only the disjoint combinations (A, B) and (B, C), we show that a transformer pre-trained with LoReTTa can handle any modality combination at inference time, including the never-seen pair (A, C) and the triplet (A, B, C). We evaluate our approach on a multimodal dataset derived from MNIST containing speech, vision, and language, as well as a real-world medical dataset containing mRNA, miRNA, and RPPA samples from TCGA. Compared to traditional pre-training methods, we observe up to a 100-point reduction in perplexity for autoregressive generation tasks and up to a 15% improvement in classification accuracy for previously unseen modality pairs during the pre-training phase.

翻译：摘要：在实践中，收集包含A和B或B和C两种配对模态的多模态数据集十分困难。而获取包含A、B、C三种对齐模态的数据集则更具挑战性。例如，某些公开医疗数据集仅包含某位患者的基因序列与显微图像，另一位患者仅有基因序列与放射影像——但没有任何数据集同时包含同一患者的显微图像与放射影像。这使得将全部模态整合至大规模预训练神经网络变得困难。我们提出LoReTTa（通过传递性和交换性预训练策略连接模态）来解决这一研究不足的问题。该自监督框架将因果掩码建模与交换律、传递律相结合，实现了模态内部及跨模态的转换。因此，它能通过A→B→C路径建模A→C关系。给定仅包含非重叠组合(A, B)和(B, C)的数据集，我们证明使用LoReTTa预训练的Transformer在推理时可处理任意模态组合，包括从未见过的配对(A, C)及三元组(A, B, C)。我们在基于MNIST构建的包含语音、视觉和语言的多模态数据集，以及来自TCGA包含mRNA、miRNA和RPPA样本的真实医疗数据集上评估了该方法。与传统的预训练方法相比，在自回归生成任务中观察到困惑度降低高达100个点，在预训练阶段未见过的模态对分类任务中准确率提升最高达15%。