We study streaming data with categorical features where the vocabulary of categorical feature values is changing and can even grow unboundedly over time. Feature hashing is commonly used as a pre-processing step to map these categorical values into a feature space of fixed size before learning their embeddings. While these methods have been developed and evaluated for offline or batch settings, in this paper we consider online settings. We show that deterministic embeddings are sensitive to the arrival order of categories and suffer from forgetting in online learning, leading to performance deterioration. To mitigate this issue, we propose a probabilistic hash embedding (PHE) model that treats hash embeddings as stochastic and applies Bayesian online learning to learn incrementally from data. Based on the structure of PHE, we derive a scalable inference algorithm to learn model parameters and infer/update the posteriors of hash embeddings and other latent variables. Our algorithm (i) can handle an evolving vocabulary of categorical items, (ii) is adaptive to new items without forgetting old items, (iii) is implementable with a bounded set of parameters that does not grow with the number of distinct observed values on the stream, and (iv) is invariant to the item arrival order. Experiments in classification, sequence modeling, and recommendation systems in online learning setups demonstrate the superior performance of PHE while maintaining high memory efficiency (consumes as low as 2~4 memory of a one-hot embedding table). Supplementary materials are at https://github.com/aodongli/probabilistic-hash-embeddings

翻译：我们研究具有类别特征的流式数据，其中类别特征值的词汇表随时间变化，甚至可能无限增长。特征哈希通常作为预处理步骤，将这些类别值映射到固定大小的特征空间，然后学习其嵌入表示。尽管这些方法已在离线或批量设置中得到开发和评估，但本文关注在线设置。我们证明确定性嵌入对类别到达顺序敏感，并在在线学习中易受遗忘影响，导致性能下降。为缓解此问题，我们提出一种概率哈希嵌入（PHE）模型，将哈希嵌入视为随机变量，并应用贝叶斯在线学习从数据中增量学习。基于PHE的结构，我们推导出一种可扩展的推断算法，以学习模型参数并推断/更新哈希嵌入及其他隐变量的后验分布。我们的算法（i）能处理不断演化的类别项词汇表，（ii）适应新项而不遗忘旧项，（iii）可通过一组有界参数实现，其数量不随流中观测到的不同值数量增长，且（iv）对项到达顺序保持不变性。在在线学习设置下的分类、序列建模和推荐系统实验表明，PHE在保持高内存效率（仅消耗独热嵌入表2~4倍内存）的同时，展现出优越性能。补充材料见 https://github.com/aodongli/probabilistic-hash-embeddings