In this paper, we explore the idea of analysing the historical bias of contextual language models based on BERT by measuring their adequacy with respect to Early Modern (EME) and Modern (ME) English. In our preliminary experiments, we perform fill-in-the-blank tests with 60 masked sentences (20 EME-specific, 20 ME-specific and 20 generic) and three different models (i.e., BERT Base, MacBERTh, English HLM). We then rate the model predictions according to a 5-point bipolar scale between the two language varieties and derive a weighted score to measure the adequacy of each model to EME and ME varieties of English.

翻译：本文探索了基于BERT的上下文语言模型在历史偏见分析中的新思路，通过衡量其对早期现代英语（EME）与现代英语（ME）的适配程度展开研究。在初步实验中，我们采用60个掩码句子（其中20个为EME特定句、20个为ME特定句、20个为通用句）及三种不同模型（即BERT Base、MacBERTh、English HLM）进行填空测试。随后依据两种语言变体间的五级双极量表对模型预测结果进行评分，并推导出加权得分以衡量各模型对EME和ME英语变体的适配程度。