After just a few hundred training updates, a standard probabilistic model for language generation has likely not yet learnt many semantic or syntactic rules of natural language, making it difficult to estimate the probability distribution over next tokens. Yet around this point, these models have identified a simple, loss-minimising behaviour: to output the unigram distribution of the target training corpus. The use of such a heuristic raises the question: Can we initialise our models with this behaviour and save precious compute resources and model capacity? Here we show that we can effectively endow standard neural language generation models with a separate module that reflects unigram frequency statistics as prior knowledge, simply by initialising the bias term in a model's final linear layer with the log-unigram distribution. We use neural machine translation as a test bed for this simple technique and observe that it: (i) improves learning efficiency; (ii) achieves better overall performance; and perhaps most importantly (iii) appears to disentangle strong frequency effects by encouraging the model to specialise in non-frequency-related aspects of language.

翻译：在仅经过数百次训练更新后，标准概率语言生成模型很可能尚未习得自然语言的诸多语义或句法规则，这使得估计下一个词元的概率分布变得困难。然而在此阶段，这些模型已识别出一种简单且能最小化损失的行为：输出目标训练语料的单字频率分布。采用这种启发式方法引出一个问题：我们能否通过此类行为初始化模型，从而节省宝贵的计算资源和模型容量？本文证明，只需将模型最终线性层中的偏置项初始化为对数单字分布，就能有效为标准神经语言生成模型赋予一个独立模块，该模块以先验知识形式反映单字频率统计特征。我们以神经机器翻译作为该简单技术的测试平台，观察到其：（i）提升学习效率；（ii）获得更优的整体性能；以及最重要的（iii）通过鼓励模型专注于语言的非频率相关方面，似乎能分离出强烈的频率效应。