Recently, the ever-increasing demand for bandwidth in multi-modal communication systems requires a paradigm shift. Powered by deep learning, semantic communications are applied to multi-modal scenarios to boost communication efficiency and save communication resources. However, the existing end-to-end neural network (NN) based framework without the channel encoder/decoder is incompatible with modern digital communication systems. Moreover, most end-to-end designs are task-specific and require re-design and re-training for new tasks, which limits their applications. In this paper, we propose a distributed multi-modal semantic communication framework incorporating the conventional channel encoder/decoder. We adopt NN-based semantic encoder and decoder to extract correlated semantic information contained in different modalities, including speech, text, and image. Based on the proposed framework, we further establish a general rate-adaptive coding mechanism for various types of multi-modal semantic tasks. In particular, we utilize unequal error protection based on semantic importance, which is derived by evaluating the distortion bound of each modality. We further formulate and solve an optimization problem that aims at minimizing inference delay while maintaining inference accuracy for semantic tasks. Numerical results show that the proposed mechanism fares better than both conventional communication and existing semantic communication systems in terms of task performance, inference delay, and deployment complexity.

翻译：近年来，多模态通信系统对带宽的持续增长需求要求范式转变。借助深度学习的赋能，语义通信被应用于多模态场景以提升通信效率并节省通信资源。然而，现有基于端到端神经网络且不含信道编解码器的框架与现代数字通信系统不兼容。此外，多数端到端设计具有任务特异性，需针对新任务重新设计与训练，这限制了其应用范围。本文提出一种融合传统信道编解码器的分布式多模态语义通信框架，采用基于神经网络的语义编解码器提取语音、文本和图像等不同模态中包含的相关语义信息。基于所提框架，我们进一步建立适用于多类型多模态语义任务的通用速率自适应编码机制，具体通过评估各模态的失真界推导语义重要性，并实现基于语义重要性的非等差错保护。我们构建并求解了以最小化推理延迟并维持语义任务推理精度为目标的优化问题。数值结果表明，所提机制在任务性能、推理延迟及部署复杂度方面均优于传统通信系统与现有语义通信系统。