Language model probing is often used to test specific capabilities of models. However, conclusions from such studies may be limited when the probing benchmarks are small and lack statistical power. In this work, we introduce new, larger datasets for negation (NEG-1500-SIMP) and role reversal (ROLE-1500) inspired by psycholinguistic studies. We dramatically extend existing NEG-136 and ROLE-88 benchmarks using GPT3, increasing their size from 18 and 44 sentence pairs to 750 each. We also create another version of extended negation dataset (NEG-1500-SIMP-TEMP), created using template-based generation. It consists of 770 sentence pairs. We evaluate 22 models on the extended datasets, seeing model performance dip 20-57% compared to the original smaller benchmarks. We observe high levels of negation sensitivity in models like BERT and ALBERT demonstrating that previous findings might have been skewed due to smaller test sets. Finally, we observe that while GPT3 has generated all the examples in ROLE-1500 is only able to solve 24.6% of them during probing. The datasets and code are available on $\href{https://github.com/text-machine-lab/extending_psycholinguistic_dataset}{Github}$.

翻译：语言模型探针常被用于测试模型的特定能力。然而，当探针评估基准数据集规模较小且缺乏统计效力时，此类研究的结论可能存在局限性。本研究受心理语言学启发，构建了新的更大规模数据集：否定数据集（NEG-1500-SIMP）和角色反转数据集（ROLE-1500）。我们采用GPT3显著扩展了现有NEG-136和ROLE-88基准数据集，将各自的句子对数量从18对和44对分别扩展至750对。此外，我们还通过模板生成方法创建了另一个扩展版否定数据集（NEG-1500-SIMP-TEMP），包含770个句子对。在扩展数据集上评估了22个模型后，发现模型性能较原小规模基准数据集下降20-57%。观察到BERT、ALBERT等模型具有较高的否定敏感度，表明先前研究结果可能因测试集规模较小而产生偏差。值得注意的是，尽管ROLE-1500中的所有示例均由GPT3生成，但该模型在探针测试中仅能解决其中24.6%的问题。数据集与代码已发布于$\href{https://github.com/text-machine-lab/extending_psycholinguistic_dataset}{Github}$。