Over the last few years, machine learning unlocked previously infeasible features for compression, such as providing guarantees for users' privacy or tailoring compression to specific data statistics (e.g., satellite images or audio recordings of animals) or users' audiovisual perception. This, in turn, has led to an explosion of theoretical investigations and insights that aim to develop new fundamental theories, methods and algorithms better suited for machine learning-based compressors. In this thesis, I contribute to this trend by investigating relative entropy coding, a mathematical framework that generalises classical source coding theory. Concretely, relative entropy coding deals with the efficient communication of uncertain or randomised information. One of its key advantages is that it extends compression methods to continuous spaces and can thus be integrated more seamlessly into modern machine learning pipelines than classical quantisation-based approaches. Furthermore, it is a natural foundation for developing advanced compression methods that are privacy-preserving or account for the perceptual quality of the reconstructed data. The thesis considers relative entropy coding at three conceptual levels: After introducing the basics of the framework, (1) I prove results that provide new, maximally tight fundamental limits to the communication and computational efficiency of relative entropy coding; (2) I use the theory of Poisson point processes to develop and analyse new relative entropy coding algorithms, whose performance attains the theoretic optima and (3) I showcase the strong practical performance of relative entropy coding by applying it to image, audio, video and protein data compression using small, energy-efficient, probabilistic neural networks called Bayesian implicit neural representations.

翻译：在过去几年中，机器学习解锁了压缩领域此前难以实现的功能，例如为用户隐私提供保障、针对特定数据统计特性（如卫星图像或动物音频记录）定制压缩方案，以及适配用户的视听感知。这进而催生了大量理论探索与洞见，旨在发展更适合基于机器学习压缩器的新型基础理论、方法与算法。在本论文中，我通过研究相对熵编码——一种泛化经典信源编码理论的数学框架——为这一趋势做出贡献。具体而言，相对熵编码处理的是不确定或随机化信息的高效通信问题。其关键优势之一在于将压缩方法扩展至连续空间，从而比基于经典量化的方法更无缝地集成到现代机器学习流水线中。此外，它也为开发保护隐私或兼顾重构数据感知质量的先进压缩方法提供了自然基础。本论文从三个概念层面审视相对熵编码：在介绍该框架的基本原理后，(1) 我证明了若干结果，为相对熵编码的通信与计算效率提供了新的、最大紧致的基本极限；(2) 我利用泊松点过程理论开发并分析了新的相对熵编码算法，其性能达到理论最优值；(3) 我通过将相对熵编码应用于图像、音频、视频及蛋白质数据压缩，使用名为贝叶斯隐式神经表示的小型、节能概率神经网络，展示了其强大的实践性能。