In recent years, Federated Learning (FL) has shown significant advancements in its ability to perform various natural language processing (NLP) tasks. This work focuses on applying personalized FL for on-device language modeling. Due to limitations of memory and latency, these models cannot support the complexity of sub-word tokenization or beam search decoding, resulting in the decision to deploy a closed-vocabulary language model. However, closed-vocabulary models are unable to handle out-of-vocabulary (OOV) words belonging to specific users. To address this issue, We propose a novel technique called "OOV expansion" that improves OOV coverage and increases model accuracy while minimizing the impact on memory and latency. This method introduces a personalized "OOV adapter" that effectively transfers knowledge from a central model and learns word embedding for personalized vocabulary. OOV expansion significantly outperforms standard FL personalization methods on a set of common FL benchmarks.

翻译：近年来，联邦学习（FL）在执行各类自然语言处理（NLP）任务方面取得了显著进展。本研究聚焦于将个性化联邦学习应用于设备端语言建模。受限于内存和延迟因素，这些模型无法支持子词分词或波束搜索解码的复杂度，因此决定部署闭词汇表语言模型。然而，闭词汇表模型无法处理属于特定用户的词汇表外（OOV）单词。为解决该问题，我们提出一种称为"OOV扩展"的新技术，该技术能在最小化对内存和延迟影响的同时，提升OOV覆盖率和模型准确率。该方法引入了个性化"OOV适配器"，它能有效迁移中央模型的知识，并为个性化词汇学习词嵌入。在一组标准联邦学习基准测试中，OOV扩展显著优于常规的联邦学习个性化方法。