The process industry's high expectations for Digital Twins require modeling approaches that can generalize across tasks and diverse domains with potentially different data dimensions and distributional shifts i.e., Foundational Models. Despite success in natural language processing and computer vision, transfer learning with (self-) supervised signals for pre-training general-purpose models is largely unexplored in the context of Digital Twins in the process industry due to challenges posed by multi-dimensional time-series data, lagged cause-effect dependencies, complex causal structures, and varying number of (exogenous) variables. We propose a novel channel-dependent pre-training strategy that leverages synchronized cause-effect pairs to overcome these challenges by breaking down the multi-dimensional time-series data into pairs of cause-effect variables. Our approach focuses on: (i) identifying highly lagged causal relationships using data-driven methods, (ii) synchronizing cause-effect pairs to generate training samples for channel-dependent pre-training, and (iii) evaluating the effectiveness of this approach in channel-dependent forecasting. Our experimental results demonstrate significant improvements in forecasting accuracy and generalization capability compared to traditional training methods.

翻译：流程工业对数字孪生的高期望要求建模方法能够跨越任务和不同领域进行泛化，这些领域可能具有不同的数据维度和分布偏移，即基础模型。尽管在自然语言处理和计算机视觉领域取得了成功，但在流程工业的数字孪生背景下，利用（自）监督信号进行预训练通用模型的迁移学习在很大程度上尚未得到探索，这主要是由于多维时间序列数据、滞后的因果依赖关系、复杂的因果结构以及（外生）变量数量变化所带来的挑战。我们提出了一种新颖的通道依赖预训练策略，通过将多维时间序列数据分解为因果变量对，利用同步的因果对来克服这些挑战。我们的方法侧重于：（i）使用数据驱动方法识别高度滞后的因果关系，（ii）同步因果对以生成用于通道依赖预训练的训练样本，以及（iii）评估该方法在通道依赖预测中的有效性。我们的实验结果表明，与传统训练方法相比，该方法在预测准确性和泛化能力方面均有显著提升。