Over the last few years, machine learning unlocked previously infeasible features for compression, such as providing guarantees for users' privacy or tailoring compression to specific data statistics (e.g., satellite images or audio recordings of animals) or users' audiovisual perception. This, in turn, has led to an explosion of theoretical investigations and insights that aim to develop new fundamental theories, methods and algorithms better suited for machine learning-based compressors. In this thesis, I contribute to this trend by investigating relative entropy coding, a mathematical framework that generalises classical source coding theory. Concretely, relative entropy coding deals with the efficient communication of uncertain or randomised information. One of its key advantages is that it extends compression methods to continuous spaces and can thus be integrated more seamlessly into modern machine learning pipelines than classical quantisation-based approaches. Furthermore, it is a natural foundation for developing advanced compression methods that are privacy-preserving or account for the perceptual quality of the reconstructed data. The thesis considers relative entropy coding at three conceptual levels: After introducing the basics of the framework, (1) I prove results that provide new, maximally tight fundamental limits to the communication and computational efficiency of relative entropy coding; (2) I use the theory of Poisson point processes to develop and analyse new relative entropy coding algorithms, whose performance attains the theoretic optima and (3) I showcase the strong practical performance of relative entropy coding by applying it to image, audio, video and protein data compression using small, energy-efficient, probabilistic neural networks called Bayesian implicit neural representations.

翻译：近年来，机器学习为数据压缩解锁了以往难以实现的功能，例如为用户隐私提供保障，或针对特定数据统计特性（如卫星图像或动物音频记录）及用户视听感知特性定制压缩方案。这进而引发了旨在发展更适用于基于机器学习的压缩器的新基础理论、方法与算法的理论探索与见解的爆发式增长。本论文通过研究相对熵编码——一种推广经典信源编码理论的数学框架——为这一趋势做出贡献。具体而言，相对熵编码处理不确定或随机化信息的高效通信。其关键优势之一在于将压缩方法扩展至连续空间，从而能比经典的基于量化的方法更无缝地集成到现代机器学习流程中。此外，它自然构成了开发具有隐私保护能力或考虑重建数据感知质量的高级压缩方法的基础。本论文在三个概念层面探讨相对熵编码：在介绍该框架基础后，（1）我证明了为相对熵编码的通信与计算效率提供新的、最大程度紧致的基本极限的结果；（2）我利用泊松点过程理论开发并分析了新的相对熵编码算法，其性能达到理论最优；（3）通过将其应用于图像、音频、视频和蛋白质数据压缩，我展示了相对熵编码的强大实际性能，所用工具为称为贝叶斯隐式神经表示的小型、高能效概率神经网络。