The concept of attention, numerical weights that emphasize the importance of particular data, has proven to be very relevant in artificial intelligence. Relative entropy (RE, aka Kullback-Leibler divergence) plays a central role in communication theory. Here we combine these concepts, attention and RE. RE guides optimal encoding of messages in bandwidth-limited communication as well as optimal message decoding via the maximum entropy principle (MEP). In the coding scenario, RE can be derived from four requirements, namely being analytical, local, proper, and calibrated. Weighted RE, used for attention steering in communications, turns out to be improper. To see how proper attention communication can emerge, we analyze a scenario of a message sender who wants to ensure that the receiver of the message can perform well-informed actions. If the receiver decodes the message using the MEP, the sender only needs to know the receiver's utility function to inform optimally, but not the receiver's initial knowledge state. In case only the curvature of the utility function maxima are known, it becomes desirable to accurately communicate an attention function, in this case a by this curvature weighted and re-normalized probability function. Entropic attention communication is here proposed as the desired generalization of entropic communication that permits weighting while being proper, thereby aiding the design of optimal communication protocols in technical applications and helping to understand human communication. For example, our analysis shows how to derive the level of cooperation expected under misaligned interests of otherwise honest communication partners.

翻译：注意力机制通过数值权重强调特定数据的重要性，已在人工智能领域展现出显著价值。相对熵（RE，即KL散度）在通信理论中占据核心地位。本文融合注意力与相对熵这两个概念：相对熵既指导带宽受限通信中的最优消息编码，也通过最大熵原理（MEP）实现最优消息解码。在编码场景下，相对熵可由四个必要条件推导得出——解析性、局部性、恰当性与校准性。用于引导通信注意力的加权相对熵被证明不满足恰当性。为探究恰当注意力通信的实现路径，我们分析了一个场景：消息发送者需确保接收方能基于消息做出充分知情决策。若接收方采用最大熵原理解码消息，发送者只需获知接收方的效用函数即可实现最优信息传递，而无需掌握其初始知识状态。当仅知效用函数极大值曲率时，需精确传递注意力函数——即经该曲率加权并重新归一化的概率函数。本文提出熵注意力通信作为熵通信的理想泛化形式，其在保持恰当性的同时引入权重机制，从而为技术应用中设计最优通信协议提供支撑，并助力理解人类通信行为。例如，我们的分析揭示了如何推导诚实通信伙伴在利益分歧情况下的预期合作程度。