The differences between cloze-task language model (LM) probing with 1) expert-made templates and 2) naturally-occurring text have often been overlooked. Here, we evaluate 16 different LMs on 10 probing English datasets -- 4 template-based and 6 template-free -- in general and biomedical domains to answer the following research questions: (RQ1) Do model rankings differ between the two approaches? (RQ2) Do models' absolute scores differ between the two approaches? (RQ3) Do the answers to RQ1 and RQ2 differ between general and domain-specific models? Our findings are: 1) Template-free and template-based approaches often rank models differently, except for the top domain-specific models. 2) Scores decrease by up to 42% Acc@1 when comparing parallel template-free and template-based prompts. 3) Perplexity is negatively correlated with accuracy in the template-free approach, but, counter-intuitively, they are positively correlated for template-based probing. 4) Models tend to predict the same answers frequently across prompts for template-based probing, which is less common when employing template-free techniques.

翻译：掩码语言模型（LM）探针研究中，采用专家构建模板与自然文本两种方式的差异常被忽视。本研究在通用领域和生物医学领域，对16种不同语言模型进行10个英文探针数据集（4个基于模板、6个无模板）的评估，旨在回答以下研究问题：（RQ1）两种方法的模型排名是否存在差异？（RQ2）两种方法的模型绝对得分是否存在差异？（RQ3）RQ1与RQ2的答案是否因通用模型与领域专用模型而异？研究发现：1）除顶尖领域专用模型外，无模板与基于模板方法对模型的排名结果常存在差异；2）对比并行无模板与基于模板提示时，模型得分（Acc@1）最高下降42%；3）无模板方法中困惑度与准确率呈负相关，但反直觉的是，基于模板探针中两者呈正相关；4）基于模板探针中模型倾向于跨提示重复预测相同答案，而无模板技术中该现象较少见。