Generative language models are usually pretrained on large text corpus via predicting the next token (i.e., sub-word/word/phrase) given the previous ones. Recent works have demonstrated the impressive performance of large generative language models on downstream tasks. However, existing generative language models generally neglect an inherent challenge in text corpus during training, i.e., the imbalance between frequent tokens and infrequent ones. It can lead a language model to be dominated by common and easy-to-learn tokens, thereby overlooking the infrequent and difficult-to-learn ones. To alleviate that, we propose a MiLe Loss function for mitigating the bias of learning difficulties with tokens. During training, it can dynamically assess the learning difficulty of a to-be-learned token, according to the information entropy of the corresponding predicted probability distribution over the vocabulary. Then it scales the training loss adaptively, trying to lead the model to focus more on the difficult-to-learn tokens. On the Pile dataset, we train generative language models at different scales of 468M, 1.2B, and 6.7B parameters. Experiments reveal that models incorporating the proposed MiLe Loss can gain consistent performance improvement on downstream benchmarks.

翻译：生成式语言模型通常通过在大规模文本语料上预测给定上文的下一个词元（即子词/词/短语）进行预训练。近期研究表明，大型生成式语言模型在下游任务中展现出令人瞩目的性能。然而，现有生成式语言模型在训练过程中普遍忽略了文本语料中的一个固有挑战——高频词元与低频词元之间的不平衡。这会导致语言模型被常见且易学习的词元所主导，从而忽视低频且难学习的词元。为解决该问题，我们提出一种名为MiLe Loss的损失函数，以缓解词元学习困难偏差。在训练过程中，该函数能根据词汇表上对应预测概率分布的信息熵，动态评估待学习词元的学习难度，并自适应地缩放训练损失，引导模型更关注难学习词元。我们在Pile数据集上训练了参数规模分别为468M、1.2B和6.7B的生成式语言模型。实验表明，采用所提出的MiLe Loss的模型在下游基准测试中均能获得一致的性能提升。