While pretrained language models (PLMs) have been shown to possess a plethora of linguistic knowledge, the existing body of research has largely neglected extralinguistic knowledge, which is generally difficult to obtain by pretraining on text alone. Here, we contribute to closing this gap by examining geolinguistic knowledge, i.e., knowledge about geographic variation in language. We introduce geoadaptation, an intermediate training step that couples language modeling with geolocation prediction in a multi-task learning setup. We geoadapt four PLMs, covering language groups from three geographic areas, and evaluate them on five different tasks: fine-tuned (i.e., supervised) geolocation prediction, zero-shot (i.e., unsupervised) geolocation prediction, fine-tuned language identification, zero-shot language identification, and zero-shot prediction of dialect features. Geoadaptation is very successful at injecting geolinguistic knowledge into the PLMs: the geoadapted PLMs consistently outperform PLMs adapted using only language modeling (by especially wide margins on zero-shot prediction tasks), and we obtain new state-of-the-art results on two benchmarks for geolocation prediction and language identification. Furthermore, we show that the effectiveness of geoadaptation stems from its ability to geographically retrofit the representation space of the PLMs.

翻译：尽管预训练语言模型（PLMs）已被证明拥有丰富的语言知识，但现有研究很大程度上忽视了语言外知识——这类知识通常难以仅通过文本预训练获得。为填补这一空白，本文考察地理语言学知识，即语言中地理变异的相关知识。我们提出地理适应（geoadaptation）方法，这是一种在多任务学习框架下将语言建模与地理位置预测相结合的中期训练步骤。我们对涵盖三个地理区域语言组的四种PLMs进行地理适应，并在五项任务上评估其表现：微调（即有监督）地理位置预测、零样本（即无监督）地理位置预测、微调语言识别、零样本语言识别以及方言特征的零样本预测。地理适应能非常成功地将地理语言学知识注入PLMs：经地理适应的PLMs始终优于仅使用语言建模适应的PLMs（在零样本预测任务上优势尤为显著），并在地理位置预测和语言识别两个基准上取得了新的最优结果。此外，我们证明地理适应的有效性源于其能够对PLMs的表示空间进行地理重构。