The transformer structure employed in large language models (LLMs), as a specialized category of deep neural networks (DNNs) featuring attention mechanisms, stands out for their ability to identify and highlight the most relevant aspects of input data. Such a capability is particularly beneficial in addressing a variety of communication challenges, notably in the realm of semantic communication where proper encoding of the relevant data is critical especially in systems with limited bandwidth. In this work, we employ vision transformers specifically for the purpose of compression and compact representation of the input image, with the goal of preserving semantic information throughout the transmission process. Through the use of the attention mechanism inherent in transformers, we create an attention mask. This mask effectively prioritizes critical segments of images for transmission, ensuring that the reconstruction phase focuses on key objects highlighted by the mask. Our methodology significantly improves the quality of semantic communication and optimizes bandwidth usage by encoding different parts of the data in accordance with their semantic information content, thus enhancing overall efficiency. We evaluate the effectiveness of our proposed framework using the TinyImageNet dataset, focusing on both reconstruction quality and accuracy. Our evaluation results demonstrate that our framework successfully preserves semantic information, even when only a fraction of the encoded data is transmitted, according to the intended compression rates.

翻译：大型语言模型（LLMs）中采用的Transformer结构，作为一类具有注意力机制的特殊深度神经网络（DNNs），因其能够识别并突出输入数据中最相关的方面而脱颖而出。这一能力在解决各种通信挑战时尤为有益，特别是在语义通信领域——在带宽受限的系统中，正确编码相关数据至关重要。在本研究中，我们专门采用视觉Transformer对输入图像进行压缩和紧凑表示，旨在传输过程中保留语义信息。通过利用Transformer固有的注意力机制，我们生成了一个注意力掩码。该掩码能有效优先传输图像的关键部分，确保重建阶段聚焦于掩码所强调的核心对象。我们的方法根据数据的不同语义信息内容对其各部分进行编码，从而显著提升语义通信质量并优化带宽使用，进而提高整体效率。我们使用TinyImageNet数据集对所提框架的有效性进行了评估，重点关注重建质量与准确性。评估结果表明，即使仅传输编码数据的一小部分（按照目标压缩率），我们的框架也能成功保留语义信息。