Many, if not most, systems of interest in science are naturally described as nonlinear dynamical systems. Empirically, we commonly access these systems through time series measurements. Often such time series may consist of discrete random variables rather than continuous measurements, or may be composed of measurements from multiple data modalities observed simultaneously. For instance, in neuroscience we may have behavioral labels in addition to spike counts and continuous physiological recordings. While by now there is a burgeoning literature on deep learning for dynamical systems reconstruction (DSR), multimodal data integration has hardly been considered in this context. Here we provide such an efficient and flexible algorithmic framework that rests on a multimodal variational autoencoder for generating a sparse teacher signal that guides training of a reconstruction model, exploiting recent advances in DSR training techniques. It enables to combine various sources of information for optimal reconstruction, even allows for reconstruction from symbolic data (class labels) alone, and connects different types of observations within a common latent dynamics space. In contrast to previous multimodal data integration techniques for scientific applications, our framework is fully \textit{generative}, producing, after training, trajectories with the same geometrical and temporal structure as those of the ground truth system.

翻译：科学领域中许多（若非绝大多数）感兴趣的系统自然地被描述为非线性动力系统。经验上，我们通常通过时间序列测量来观测这些系统。这类时间序列常由离散随机变量而非连续测量值构成，或可能包含同时观测的多种数据模态的测量结果。例如在神经科学中，我们除了尖峰计数和连续生理记录外，还可能获得行为标签数据。尽管目前关于深度学习用于动力系统重构（DSR）的研究文献日益丰富，但多模态数据整合在此背景下却鲜有探讨。本文提出一种高效灵活的算法框架，其核心在于利用多模态变分自编码器生成稀疏教师信号，以指导重构模型的训练，并融合了DSR训练技术的最新进展。该框架能够整合多种信息源以实现最优重构，甚至支持仅从符号数据（类别标签）进行重构，并将不同类型的观测数据关联到共同的潜在动力学空间中。与先前科学应用中的多模态数据整合技术相比，我们的框架具有完全的\textit{生成能力}，训练后能够产生与真实系统具有相同几何结构和时间特征的轨迹。