We present Bit Diffusion: a simple and generic approach for generating discrete data with continuous state and continuous time diffusion models. The main idea behind our approach is to first represent the discrete data as binary bits, and then train a continuous diffusion model to model these bits as real numbers which we call analog bits. To generate samples, the model first generates the analog bits, which are then thresholded to obtain the bits that represent the discrete variables. We further propose two simple techniques, namely Self-Conditioning and Asymmetric Time Intervals, which lead to a significant improvement in sample quality. Despite its simplicity, the proposed approach can achieve strong performance in both discrete image generation and image captioning tasks. For discrete image generation, we significantly improve previous state-of-the-art on both CIFAR-10 (which has 3K discrete 8-bit tokens) and ImageNet-64x64 (which has 12K discrete 8-bit tokens), outperforming the best autoregressive model in both sample quality (measured by FID) and efficiency. For image captioning on MS-COCO dataset, our approach achieves competitive results compared to autoregressive models.

翻译：我们提出位扩散（Bit Diffusion）方法：一种利用连续状态和连续时间扩散模型生成离散数据的简单通用方法。该方案的核心思想是将离散数据首先表示为二进制位，随后训练连续扩散模型将这些位建模为称为"模拟比特"的实数。生成样本时，模型先生成模拟比特，再通过阈值化获取表征离散变量的二进制位。我们进一步提出两项简单技术——自条件机制（Self-Conditioning）与非对称时间间隔（Asymmetric Time Intervals）——显著提升了样本质量。尽管方法简洁，该方案在离散图像生成和图像描述任务中均展现强劲性能。在离散图像生成方面，我们显著改进了CIFAR-10（含3K离散8位令牌）和ImageNet-64x64（含12K离散8位令牌）的先前最佳结果，在样本质量（以FID为指标）和效率上均超越最优自回归模型。在MS-COCO数据集上的图像描述任务中，我们的方法达到了与自回归模型相媲美的性能。