This paper discusses the limitations of evaluating Masked Language Models (MLMs) in code completion tasks. We highlight that relying on accuracy-based measurements may lead to an overestimation of models' capabilities by neglecting the syntax rules of programming languages. To address these issues, we introduce a technique called SyntaxEval in which Syntactic Capabilities are used to enhance the evaluation of MLMs. SyntaxEval automates the process of masking elements in the model input based on their Abstract Syntax Trees (ASTs). We conducted a case study on two popular MLMs using data from GitHub repositories. Our results showed negative causal effects between the node types and MLMs' accuracy. We conclude that MLMs under study fail to predict some syntactic capabilities.

翻译：本文探讨了在代码补全任务中评估掩码语言模型（MLMs）的局限性。我们指出，依赖基于准确率的度量可能因忽视编程语言的语法规则而高估模型的能力。为解决这些问题，我们引入了一种名为SyntaxEval的技术，该技术利用句法能力增强对MLMs的评估。SyntaxEval基于输入的抽象语法树（ASTs）自动实现了模型输入中元素的掩码过程。我们利用GitHub仓库的数据对两种主流MLMs进行了案例研究。结果表明，节点类型与MLMs的准确率之间存在负向因果效应。我们得出结论，所研究的MLMs未能预测某些句法能力。