Recent psycholinguistic studies have drawn conflicting conclusions about the relationship between the quality of a language model and the ability of its surprisal estimates to predict human reading times, which has been speculated to be due to the large gap in both the amount of training data and model capacity across studies. The current work aims to consolidate these findings by evaluating surprisal estimates from Transformer-based language model variants that vary systematically in the amount of training data and model capacity on their ability to predict human reading times. The results show that surprisal estimates from most variants with contemporary model capacities provide the best fit after seeing about two billion training tokens, after which they begin to diverge from humanlike expectations. Additionally, newly-trained smaller model variants reveal a 'tipping point' at convergence, after which the decrease in language model perplexity begins to result in poorer fits to human reading times. These results suggest that the massive amount of training data is mainly responsible for the poorer fit achieved by surprisal from larger pre-trained language models, and that a certain degree of model capacity is necessary for Transformer-based language models to capture humanlike expectations.

翻译：近期心理语言学研究中，关于语言模型质量与其惊异度估计预测人类阅读时间能力之间的关系，得出了相互矛盾的结论。这些矛盾被推测源于不同研究在训练数据量和模型能力上的巨大差异。本研究旨在通过评估基于Transformer的语言模型变体（这些变体在训练数据量和模型能力上系统性地变化）的惊异度估计在预测人类阅读时间方面的能力，来整合这些发现。结果表明，在当代模型能力下，大多数变体的惊异度估计在观察到约二十亿训练词元后达到最佳拟合，此后开始偏离人类化预期。此外，新训练的较小模型变体在收敛点揭示了一个“临界点”，之后语言模型困惑度的降低反而导致对人类阅读时间的拟合变差。这些结果表明，大量训练数据是导致较大预训练语言模型惊异度拟合效果较差的主要原因，而Transformer语言模型需要一定程度的模型能力才能捕捉人类化预期。