We investigate how pretrained language models (PLM) encode the grammatical category of verbal aspect in Russian. Encoding of aspect in transformer LMs has not been studied previously in any language. A particular challenge is posed by "alternative contexts": where either the perfective or the imperfective aspect is suitable grammatically and semantically. We perform probing using BERT and RoBERTa on alternative and non-alternative contexts. First, we assess the models' performance on aspect prediction, via behavioral probing. Next, we examine the models' performance when their contextual representations are substituted with counterfactual representations, via causal probing. These counterfactuals alter the value of the "boundedness" feature--a semantic feature, which characterizes the action in the context. Experiments show that BERT and RoBERTa do encode aspect--mostly in their final layers. The counterfactual interventions affect perfective and imperfective in opposite ways, which is consistent with grammar: perfective is positively affected by adding the meaning of boundedness, and vice versa. The practical implications of our probing results are that fine-tuning only the last layers of BERT on predicting aspect is faster and more effective than fine-tuning the whole model. The model has high predictive uncertainty about aspect in alternative contexts, which tend to lack explicit hints about the boundedness of the described action.

翻译：本研究旨在探究预训练语言模型（PLM）如何编码俄语中动词的体范畴。此前尚未有研究针对任何语言探讨Transformer语言模型中的体范畴编码问题。研究面临的一个特殊挑战在于“可替换语境”：即完成体或未完成体在语法和语义上均可适用的语境。我们基于BERT和RoBERTa模型，在可替换及非可替换语境中进行了探测分析。首先，通过行为探测评估模型在体范畴预测任务上的表现；其次，通过因果探测方法，将模型的上下文表征替换为反事实表征，考察其性能变化。这些反事实表征改变了“有界性”特征的值——该语义特征用于刻画语境中动作的属性。实验表明，BERT和RoBERTa确实编码了体范畴信息，且主要分布于模型最后几层。反事实干预对完成体和未完成体产生相反方向的影响，这与语法规律一致：增加有界性含义对完成体产生正向影响，反之则对未完成体产生正向影响。本探测研究的实践意义在于：仅对BERT最后几层进行体范畴预测的微调，相比微调整个模型更为高效且效果更优。模型在可替换语境中对体范畴的预测具有较高不确定性，这类语境往往缺乏对所述动作有界性的显式提示。