This paper discusses the limitations of evaluating Masked Language Models (MLMs) in code completion tasks. We highlight that relying on accuracy-based measurements may lead to an overestimation of models' capabilities by neglecting the syntax rules of programming languages. To address these issues, we introduce a technique called SyntaxEval in which Syntactic Capabilities are used to enhance the evaluation of MLMs. SyntaxEval automates the process of masking elements in the model input based on their Abstract Syntax Trees (ASTs). We conducted a case study on two popular MLMs using data from GitHub repositories. Our results showed negative causal effects between the node types and MLMs' accuracy. We conclude that MLMs under study fail to predict some syntactic capabilities.

翻译：本文探讨了遮蔽语言模型（MLMs）在代码补全任务中评估方法的局限性。我们指出，单纯依赖准确率指标可能因忽略编程语言语法规则而高估模型能力。针对这一问题，提出了一种名为SyntaxEval的技术，通过语法能力增强MLMs评估效果。SyntaxEval基于抽象语法树（ASTs）实现模型输入元素的自动化遮蔽处理。我们以GitHub仓库数据为基础，对两种主流MLMs进行了案例研究。结果表明，节点类型与MLMs准确率之间存在负向因果关系。据此得出，研究所涉MLMs未能有效预测部分语法能力。