Locality-sensitive hashing (LSH) based frameworks have been used efficiently to select weight vectors in a dense hidden layer with high cosine similarity to an input, enabling dynamic pruning. While this type of scheme has been shown to improve computational training efficiency, existing algorithms require repeated randomized projection of the full layer weight, which is impractical for computational- and memory-constrained devices. In a distributed setting, deferring LSH analysis to a centralized host is (i) slow if the device cluster is large and (ii) requires access to input data which is forbidden in a federated context. Using a new family of hash functions, we develop one of the first private, personalized, and memory-efficient on-device LSH frameworks. Our framework enables privacy and personalization by allowing each device to generate hash tables, without the help of a central host, using device-specific hashing hyper-parameters (e.g. number of hash tables or hash length). Hash tables are generated with a compressed set of the full weights, and can be serially generated and discarded if the process is memory-intensive. This allows devices to avoid maintaining (i) the fully-sized model and (ii) large amounts of hash tables in local memory for LSH analysis. We prove several statistical and sensitivity properties of our hash functions, and experimentally demonstrate that our framework is competitive in training large-scale recommender networks compared to other LSH frameworks which assume unrestricted on-device capacity.

翻译：基于局部敏感哈希（LSH）的框架已被有效用于选择与输入具有高余弦相似度的密集隐藏层中的权重向量，从而实现动态剪枝。虽然这类方案已被证明能提升计算训练效率，但现有算法需要对全层权重进行重复随机投影，这对于计算和内存受限的设备而言不切实际。在分布式环境中，将LSH分析集中到中心主机存在以下问题：(i) 当设备集群规模较大时处理速度缓慢；(ii) 需要访问输入数据，这在联邦学习场景中是被禁止的。通过采用新型哈希函数族，我们开发了首批兼具隐私性、个性化与内存高效性的设备本地LSH框架之一。该框架允许各设备在无需中心主机协助的情况下，通过设备特定的哈希超参数（如哈希表数量或哈希长度）生成哈希表，从而实现隐私保护与个性化。哈希表基于权重完整集合的压缩版本生成，且可在内存密集型过程中串行生成并丢弃。这使得设备无需在本地内存中维护(i)完整尺寸模型和(ii)大量用于LSH分析的哈希表。我们证明了所提哈希函数的统计特性与敏感性特征，并通过实验表明：与假设设备容量不受限的其他LSH框架相比，本框架在大规模推荐网络训练中具有竞争力。