Transfer entropy (TE) is a measurement in information theory that reveals the directional flow of information between processes, providing valuable insights for a wide range of real-world applications. This work proposes Transfer Entropy Estimation via Transformers (TREET), a novel transformer-based approach for estimating the TE for stationary processes. The proposed approach employs Donsker-Vardhan (DV) representation to TE and leverages the attention mechanism for the task of neural estimation. We propose a detailed theoretical and empirical study of the TREET, comparing it to existing methods. To increase its applicability, we design an estimated TE optimization scheme that is motivated by the functional representation lemma. Afterwards, we take advantage of the joint optimization scheme to optimize the capacity of communication channels with memory, which is a canonical optimization problem in information theory, and show the memory capabilities of our estimator. Finally, we apply TREET to real-world feature analysis. Our work, applied with state-of-the-art deep learning methods, opens a new door for communication problems which are yet to be solved.

翻译：转移熵（Transfer Entropy, TE）是信息论中一种度量指标，用于揭示过程间信息的方向性流动，为广泛的实际应用提供了重要见解。本文提出基于Transformer的转移熵估计方法（TREET），这是一种基于Transformer的创新方法，用于估计平稳过程的转移熵。该方法采用Donsker-Vardhan（DV）表示表征转移熵，并利用注意力机制完成神经估计任务。我们从理论和实证两个层面对TREET进行详细研究，并与现有方法进行比较。为提升其适用性，我们基于函数表示引理设计了一种估计TE的优化方案。随后，利用联合优化方案优化具有记忆的有噪信道容量——这是信息论中的经典优化问题，并展示了所提估计器的记忆能力。最后，我们将TREET应用于真实世界的特征分析。本工作结合当前最先进的深度学习方法，为尚未解决的信道通信问题开辟了新路径。