High-efficient image compression is a critical requirement. In several scenarios where multiple modalities of data are captured by different sensors, the auxiliary information from other modalities are not fully leveraged by existing image-only codecs, leading to suboptimal compression efficiency. In this paper, we increase image compression performance with the assistance of point cloud, which is widely adopted in the area of autonomous driving. We first unify the data representation for both modalities to facilitate data processing. Then, we propose the point cloud-assisted neural image codec (PCA-NIC) to enhance the preservation of image texture and structure by utilizing the high-dimensional point cloud information. We further introduce a multi-modal feature fusion transform module (MMFFT) to capture more representative image features, remove redundant information between channels and modalities that are not relevant to the image content. Our work is the first to improve image compression performance using point cloud and achieves state-of-the-art performance.

翻译：高效的图像压缩是一项关键需求。在多种传感器捕获多模态数据的若干场景中，现有纯图像编解码器未能充分利用来自其他模态的辅助信息，导致压缩效率欠佳。本文通过广泛用于自动驾驶领域的点云辅助，提升了图像压缩性能。我们首先统一了两种模态的数据表示以方便数据处理。随后，我们提出了点云辅助神经图像编解码器（PCA-NIC），通过利用高维点云信息来增强图像纹理和结构的保持。我们进一步引入了多模态特征融合变换模块（MMFFT），以捕获更具代表性的图像特征，并移除与图像内容无关的通道间和模态间的冗余信息。我们的工作是首个利用点云提升图像压缩性能的研究，并实现了最先进的性能。