Language model probing is often used to test specific capabilities of models. However, conclusions from such studies may be limited when the probing benchmarks are small and lack statistical power. In this work, we introduce new, larger datasets for negation (NEG-1500-SIMP) and role reversal (ROLE-1500) inspired by psycholinguistic studies. We dramatically extend existing NEG-136 and ROLE-88 benchmarks using GPT3, increasing their size from 18 and 44 sentence pairs to 750 each. We also create another version of extended negation dataset (NEG-1500-SIMP-TEMP), created using template-based generation. It consists of 770 sentence pairs. We evaluate 22 models on the extended datasets, seeing model performance dip 20-57% compared to the original smaller benchmarks. We observe high levels of negation sensitivity in models like BERT and ALBERT demonstrating that previous findings might have been skewed due to smaller test sets. Finally, we observe that while GPT3 has generated all the examples in ROLE-1500 is only able to solve 24.6% of them during probing. The datasets and code are available on $\href{https://github.com/text-machine-lab/extending_psycholinguistic_dataset}{Github}$.

翻译：语言模型探针常用于测试模型的特定能力。然而，当探针基准数据集规模较小且缺乏统计效力时，此类研究的结论可能具有局限性。本研究受心理语言学启发，引入了新的、更大的否定（NEG-1500-SIMP）和角色反转（ROLE-1500）数据集。我们利用GPT3显著扩展了现有的NEG-136和ROLE-88基准数据集，将其规模分别从18和44个句子对增加至各750个句子对。我们还通过模板生成方法创建了扩展否定数据集的另一个版本（NEG-1500-SIMP-TEMP），包含770个句子对。我们在扩展数据集上评估了22个模型，发现模型性能较原始小型基准数据集下降20-57%。在BERT和ALBERT等模型中观察到了较高的否定敏感性，这表明先前的研究结果可能因测试集规模较小而产生偏差。最后，我们发现尽管GPT3生成了ROLE-1500中的所有示例，但在探针测试中仅能正确解答其中24.6%的示例。数据集和代码已发布于$\href{https://github.com/text-machine-lab/extending_psycholinguistic_dataset}{GitHub}$。