One-hot vectors, a common method for representing discrete/categorical data, in machine learning are widely used because of their simplicity and intuitiveness. However, one-hot vectors suffer from a linear increase in dimensionality, posing computational and memory challenges, especially when dealing with datasets containing numerous categories. In this paper, we focus on tabular data generation, and reveal the multinomial diffusion faces the mode collapse phenomenon when the cardinality is high. Moreover, due to the limitations of one-hot vectors, the training phase takes time longer in such a situation. To address these issues, we propose Residual Bit Vectors (ResBit), a technique for densely representing categorical data. ResBit is an extension of analog bits and overcomes limitations of analog bits when applied to tabular data generation. Our experiments demonstrate that ResBit not only accelerates training but also maintains performance when compared with the situations before applying ResBit. Furthermore, our results indicate that many existing methods struggle with high-cardinality data, underscoring the need for lower-dimensional representations, such as ResBit and latent vectors.

翻译：独热向量作为机器学习中表示离散/分类数据的常用方法，因其简单直观而被广泛使用。然而，独热向量存在维度线性增长的问题，带来了计算和内存方面的挑战，尤其是在处理包含大量类别的数据集时。本文聚焦于表格数据生成，揭示了当基数较高时，多项扩散面临模式崩溃现象。此外，由于独热向量的局限性，在此类情况下训练阶段耗时更长。为解决这些问题，我们提出了残差位向量（ResBit），一种用于密集表示分类数据的技术。ResBit 是模拟位向量的扩展，并克服了模拟位向量在表格数据生成应用中的局限性。实验表明，与未应用 ResBit 的情况相比，ResBit 不仅加速了训练过程，同时保持了性能。此外，我们的结果表明许多现有方法难以处理高基数数据，这凸显了对低维表示（如 ResBit 和潜在向量）的需求。