Embedding tables are used by machine learning systems to work with categorical features. These tables can become exceedingly large in modern recommendation systems, necessitating the development of new methods for fitting them in memory, even during training. The best previous methods for table compression are so called "post training" quantization schemes such as "product" and "residual" quantization (Gray & Neuhoff, 1998). These methods replace table rows with references to k-means clustered "codewords". Unfortunately, clustering requires prior knowledge of the table to be compressed, which limits the memory savings to inference time and not training time. Hence, recent work, like the QR method (Shi et al., 2020), has used random references (linear sketching), which can be computed with hash functions before training. Unfortunately, the compression achieved is inferior to that achieved by post-training quantization. The new algorithm, CQR, shows how to get the best of two worlds by combining clustering and sketching: First IDs are randomly assigned to a codebook and codewords are trained (end to end) for an epoch. Next, we expand the codebook and apply clustering to reduce the size again. Finally, we add new random references and continue training. We show experimentally close to those of post-training quantization with the training time memory reductions of sketch-based methods, and we prove that our method always converges to the optimal embedding table for least-squares training.

翻译：嵌入表被机器学习系统用于处理类别型特征。在现代推荐系统中，这些表格可能会变得极其庞大，因此需要开发新方法将其存入内存，甚至在训练过程中也是如此。先前最佳的表格压缩方法被称为"训练后"量化方案，例如"乘积"和"残差"量化（Gray & Neuhoff, 1998）。这些方法将表格行替换为对k-means聚类"码字"的引用。不幸的是，聚类需要预先知道待压缩表格的信息，这限制了内存节省仅限于推理阶段，而非训练阶段。因此，近期的工作（如QR方法，Shi等，2020）采用了随机引用（线性草图），可以在训练前通过哈希函数计算。遗憾的是，其压缩效果劣于训练后量化。新算法CQR展示了如何通过结合聚类和草图来兼得两者优势：首先将ID随机分配给码本，并端到端地训练码字一个周期。随后，我们扩展码本并应用聚类以再次缩小其大小。最后，我们添加新的随机引用并继续训练。实验表明，该方法在训练时内存减少方面接近基于草图的方法，同时能达到接近训练后量化的性能，并且我们证明该方法在最小二乘训练中总能收敛到最优嵌入表。