Many, if not most, systems of interest in science are naturally described as nonlinear dynamical systems. Empirically, we commonly access these systems through time series measurements. Often such time series may consist of discrete random variables rather than continuous measurements, or may be composed of measurements from multiple data modalities observed simultaneously. For instance, in neuroscience we may have behavioral labels in addition to spike counts and continuous physiological recordings. While by now there is a burgeoning literature on deep learning for dynamical systems reconstruction (DSR), multimodal data integration has hardly been considered in this context. Here we provide such an efficient and flexible algorithmic framework that rests on a multimodal variational autoencoder for generating a sparse teacher signal that guides training of a reconstruction model, exploiting recent advances in DSR training techniques. It enables to combine various sources of information for optimal reconstruction, even allows for reconstruction from symbolic data (class labels) alone, and connects different types of observations within a common latent dynamics space. In contrast to previous multimodal data integration techniques for scientific applications, our framework is fully \textit{generative}, producing, after training, trajectories with the same geometrical and temporal structure as those of the ground truth system.

翻译：许多（即便非大多数）科学领域中关注的系统自然地描述为非线性动力系统。在经验研究中，我们通常通过时间序列测量来访问这些系统。这类时间序列有时由离散随机变量而非连续测量构成，或由同时观测的多种数据模态的测量组成。例如，在神经科学中，除了脉冲计数和连续生理记录外，我们可能还拥有行为标签。尽管目前关于深度学习在动力学系统重建（DSR）方面的文献不断增长，但多模态数据集成在该背景下几乎未被考虑。在此，我们提出一个高效且灵活的算法框架，该框架基于多模态变分自编码器生成稀疏教师信号，以引导重建模型的训练，并利用了DSR训练技术的最新进展。它能够整合多种信息源以实现最优重建，甚至仅从符号数据（如类别标签）进行重建，并将不同类型的观测连接到一个共同的潜在动力学空间中。与以往面向科学应用的多模态数据集成技术不同，我们的框架完全是**生成式**的，在训练后能够产生与真实系统具有相同几何与时间结构的轨迹。