A relative entropy code for a source $X \sim P_X$ is a stochastic code that encodes random samples from a prescribed $P_{Y \mid X}$ using as few bits as possible. A generalisation of entropy coding, it is a standard result that the minimum number of bits required to achieve this is at least the mutual information $I[X\,\Vert\,Y]$. However, a particularly fascinating feature of relative entropy coding compared to entropy coding is that, in general, this lower bound is only achievable to within an additional logarithmic factor. As such, an important research direction is to identify channels where we can reduce this gap. Sriramu and Wagner achieved such success by exhibiting a relative entropy code for so-called singular channels with sub-logarithmic asymptotic redundancy. However, their code is quite involved and, sadly, cannot be implemented in practice. In this paper, we construct the bits-back rejection sampler (BBRS), a relative entropy code that combines ideas from bits-back coding and (greedy) rejection sampling. Our analysis of BBRS reveals that the algorithm achieves the same asymptotic efficiency as Sriramu and Wagner's sampler, but with much simpler analysis and better constants. Moreover, BBRS can be implemented using standard relative entropy coding methods.

翻译：对于源$X \sim P_X$，相对熵码是一种随机码，它用尽可能少的比特数对遵循指定条件分布$P_{Y \mid X}$的随机样本进行编码。作为熵编码的推广，一个标准结论是实现此编码所需的最小比特数至少为互信息$I[X\,\Vert\,Y]$。然而，与熵编码相比，相对熵编码一个特别引人注目的特征是，该下界在一般情况下只能达到一个额外的对数因子精度。因此，一个重要研究方向是识别能够缩小这一差距的信道。Sriramu和Wagner通过展示一种针对所谓奇异信道的相对熵码（具有次对数渐近冗余度）取得了成功。但他们的编码方案相当复杂，且遗憾的是无法在实践中实现。本文构造了比特回退拒绝采样器（BBRS），这是一种结合了比特回退编码与（贪婪）拒绝采样思想的相对熵码。我们对BBRS的分析表明，该算法实现了与Sriramu和Wagner采样器相同的渐近效率，但分析更简洁且常数更优。此外，BBRS可通过标准相对熵编码方法实现。