Limited availability of labeled data for machine learning on multimodal time-series extensively hampers progress in the field. Self-supervised learning (SSL) is a promising approach to learning data representations without relying on labels. However, existing SSL methods require expensive computations of negative pairs and are typically designed for single modalities, which limits their versatility. We introduce CroSSL (Cross-modal SSL), which puts forward two novel concepts: masking intermediate embeddings produced by modality-specific encoders, and their aggregation into a global embedding through a cross-modal aggregator that can be fed to down-stream classifiers. CroSSL allows for handling missing modalities and end-to-end cross-modal learning without requiring prior data preprocessing for handling missing inputs or negative-pair sampling for contrastive learning. We evaluate our method on a wide range of data, including motion sensors such as accelerometers or gyroscopes and biosignals (heart rate, electroencephalograms, electromyograms, electrooculograms, and electrodermal) to investigate the impact of masking ratios and masking strategies for various data types and the robustness of the learned representations to missing data. Overall, CroSSL outperforms previous SSL and supervised benchmarks using minimal labeled data, and also sheds light on how latent masking can improve cross-modal learning. Our code is open-sourced at https://github.com/dr-bell/CroSSL.

翻译：摘要：多模态时间序列机器学习中标注数据的有限可用性严重阻碍了该领域的进展。自监督学习（SSL）是一种无需依赖标签即可学习数据表示的有前景方法。然而，现有的SSL方法需要昂贵的负样本对计算，且通常专门针对单一模态设计，这限制了其通用性。我们提出了CroSSL（跨模态自监督学习），引入了两个新颖概念：对模态特定编码器生成的中间嵌入进行掩蔽，以及通过跨模态聚合器将这些嵌入聚合为全局嵌入供下游分类器使用。CroSSL支持处理缺失模态和端到端的跨模态学习，无需对缺失输入进行预数据预处理或为对比学习采样负样本对。我们在一系列广泛的数据上评估了方法，包括加速度计、陀螺仪等运动传感器以及生物信号（心率、脑电图、肌电图、眼电图和皮肤电信号），以探究掩蔽比例和掩蔽策略对不同数据类型的性能影响以及学习表示对缺失数据的鲁棒性。总体而言，CroSSL在使用极少量标注数据的情况下优于先前SSL和监督学习基准，并揭示了潜在掩蔽如何改善跨模态学习。我们的代码已在https://github.com/dr-bell/CroSSL开源。