In recent years, Federated Learning (FL) has shown significant advancements in its ability to perform various natural language processing (NLP) tasks. This work focuses on applying personalized FL for on-device language modeling. Due to limitations of memory and latency, these models cannot support the complexity of sub-word tokenization or beam search decoding, resulting in the decision to deploy a closed-vocabulary language model. However, closed-vocabulary models are unable to handle out-of-vocabulary (OOV) words belonging to specific users. To address this issue, We propose a novel technique called "OOV expansion" that improves OOV coverage and increases model accuracy while minimizing the impact on memory and latency. This method introduces a personalized "OOV adapter" that effectively transfers knowledge from a central model and learns word embedding for personalized vocabulary. OOV expansion significantly outperforms standard FL personalization methods on a set of common FL benchmarks.

翻译：摘要：近年来，联邦学习（FL）在执行各种自然语言处理（NLP）任务方面展现出显著进步。本文聚焦于应用个性化联邦学习进行设备端语言建模。由于内存和延迟的限制，这些模型无法支持子词分词或集束搜索解码的复杂性，因此决定采用闭词汇语言模型。然而，闭词汇模型无法处理属于特定用户的词汇表外（OOV）单词。为解决此问题，我们提出一种名为“OOV扩展”的新技术，该技术能在最小化对内存和延迟影响的同时，提升OOV覆盖率并提高模型准确性。该方法引入了一种个性化的“OOV适配器”，可有效迁移中心模型的知识，并为个性化词汇学习词嵌入。在一系列常见FL基准测试中，OOV扩展显著优于标准FL个性化方法。