Generative modeling of non-negative, discrete data, such as symbolic music, remains challenging due to two persistent limitations in existing methods. Firstly, many approaches rely on modeling continuous embeddings, which is suboptimal for inherently discrete data distributions. Secondly, most models optimize variational bounds rather than exact data likelihood, resulting in inaccurate likelihood estimates and degraded sampling quality. While recent diffusion-based models have addressed these issues separately, we tackle them jointly. In this work, we introduce the Information-Theoretic Discrete Poisson Diffusion Model (ItDPDM), inspired by photon arrival process, which combines exact likelihood estimation with fully discrete-state modeling. Central to our approach is an information-theoretic Poisson Reconstruction Loss (PRL) that has a provable exact relationship with the true data likelihood. ItDPDM achieves improved likelihood and sampling performance over prior discrete and continuous diffusion models on a variety of synthetic discrete datasets. Furthermore, on real-world datasets such as symbolic music and images, ItDPDM attains superior likelihood estimates and competitive generation quality-demonstrating a proof of concept for distribution-robust discrete generative modeling.

翻译：非负离散数据（如符号音乐）的生成建模仍面临挑战，这主要源于现有方法的两个持续存在的局限性。首先，许多方法依赖于对连续嵌入的建模，这对于本质离散的数据分布而言并非最优。其次，大多数模型优化的是变分下界而非精确的数据似然，导致似然估计不准确并降低了采样质量。尽管最近的基于扩散的模型已分别解决了这些问题，但我们对其进行了联合处理。在本工作中，我们受光子到达过程启发，引入了信息论离散泊松扩散模型，该模型将精确似然估计与完全离散状态建模相结合。我们方法的核心是一个信息论泊松重构损失，该损失与真实数据似然存在可证明的精确关系。在多种合成离散数据集上，ItDPDM 相较于先前的离散和连续扩散模型，实现了改进的似然和采样性能。此外，在真实世界数据集（如符号音乐和图像）上，ItDPDM 获得了更优的似然估计和具有竞争力的生成质量，这为分布鲁棒的离散生成建模提供了一个概念验证。