This paper proposes a novel contrastive learning framework, called FOCAL, for extracting comprehensive features from multimodal time-series sensing signals through self-supervised training. Existing multimodal contrastive frameworks mostly rely on the shared information between sensory modalities, but do not explicitly consider the exclusive modality information that could be critical to understanding the underlying sensing physics. Besides, contrastive frameworks for time series have not handled the temporal information locality appropriately. FOCAL solves these challenges by making the following contributions: First, given multimodal time series, it encodes each modality into a factorized latent space consisting of shared features and private features that are orthogonal to each other. The shared space emphasizes feature patterns consistent across sensory modalities through a modal-matching objective. In contrast, the private space extracts modality-exclusive information through a transformation-invariant objective. Second, we propose a temporal structural constraint for modality features, such that the average distance between temporally neighboring samples is no larger than that of temporally distant samples. Extensive evaluations are performed on four multimodal sensing datasets with two backbone encoders and two classifiers to demonstrate the superiority of FOCAL. It consistently outperforms the state-of-the-art baselines in downstream tasks with a clear margin, under different ratios of available labels. The code and self-collected dataset are available at https://github.com/tomoyoshki/focal.

翻译：摘要：本文提出一种名为FOCAL的新型对比学习框架，旨在通过自监督训练从多模态时序传感信号中提取全面特征。现有大多数多模态对比框架依赖于传感模态间的共享信息，但未能明确考虑对理解底层传感物理机制至关重要的专属模态信息。此外，针对时间序列的对比框架未能恰当处理时序信息的局部性。FOCAL通过以下贡献解决上述挑战：首先，针对多模态时间序列，将每种模态编码为相互正交的共享特征与私有特征构成的因子化潜在空间。共享空间通过模态匹配目标强调跨传感模态一致的模态特征模式，而私有空间则通过变换不变性目标提取模态专属信息。其次，我们提出模态特征的时序结构约束，使得时间邻近样本间的平均距离不大于时间远距离样本间的距离。我们使用两种骨干编码器和两种分类器在四个多模态传感数据集上进行了广泛评估，证明了FOCAL的优越性。在不同标注比例的下游任务中，FOCAL均以显著优势持续超越现有最优基线方法。相关代码及自采集数据集已开源至https://github.com/tomoyoshki/focal。