Recent work has proposed explicitly inducing language-wise modularity in multilingual LMs via sparse fine-tuning (SFT) on per-language subnetworks as a means of better guiding cross-lingual sharing. In this work, we investigate (1) the degree to which language-wise modularity naturally arises within models with no special modularity interventions, and (2) how cross-lingual sharing and interference differ between such models and those with explicit SFT-guided subnetwork modularity. To quantify language specialization and cross-lingual interaction, we use a Training Data Attribution method that estimates the degree to which a model's predictions are influenced by in-language or cross-language training examples. Our results show that language-specialized subnetworks do naturally arise, and that SFT, rather than always increasing modularity, can decrease language specialization of subnetworks in favor of more cross-lingual sharing.

翻译：近期研究提出通过稀疏微调（SFT）在每种语言的子网络上显式诱导多语言语言模型的模块化特性，以更好地引导跨语言共享。本研究探讨以下两个问题：（1）未采用特殊模块化干预的模型中，语言层面模块化特性自然产生的程度；（2）此类模型与通过SFT显式引导子网络模块化的模型在跨语言共享与干扰方面的差异。为量化语言专用性与跨语言交互，我们采用训练数据归因方法，该方法可估算模型预测受同语言或跨语言训练样本影响的程度。结果表明：语言专用子网络确实会自然涌现，而SFT并非总是增强模块化，反而可能降低子网络的语言专用性，促进更广泛的跨语言共享。