Handling implicit language is essential for natural language processing systems to achieve precise text understanding and facilitate natural interactions with users. Despite its importance, the absence of a metric for accurately measuring the implicitness of language significantly constrains the depth of analysis possible in evaluating models' comprehension capabilities. This paper addresses this gap by developing a scalar metric that quantifies the implicitness level of language without relying on external references. Drawing on principles from traditional linguistics, we define ''implicitness'' as the divergence between semantic meaning and pragmatic interpretation. To operationalize this definition, we introduce ImpScore, a novel, reference-free metric formulated through an interpretable regression model. This model is trained using pairwise contrastive learning on a specially curated dataset comprising $112,580$ (implicit sentence, explicit sentence) pairs. We validate ImpScore through a user study that compares its assessments with human evaluations on out-of-distribution data, demonstrating its accuracy and strong correlation with human judgments. Additionally, we apply ImpScore to hate speech detection datasets, illustrating its utility and highlighting significant limitations in current large language models' ability to understand highly implicit content.

翻译：处理隐含语言对于自然语言处理系统实现精确文本理解并促进与用户的自然交互至关重要。尽管其重要性不言而喻，但由于缺乏能够准确衡量语言隐含程度的度量指标，在评估模型理解能力的深度分析方面受到了显著限制。本文通过开发一种不依赖外部参考的标量度量指标来量化语言的隐含程度，从而填补了这一空白。借鉴传统语言学的原理，我们将“隐含性”定义为语义含义与语用解读之间的差异。为实现这一定义的可操作化，我们引入了ImpScore，这是一种通过可解释回归模型构建的新型无参考度量指标。该模型使用配对对比学习方法，在一个专门构建的包含$112,580$对（隐含句子，显式句子）的数据集上进行训练。我们通过一项用户研究验证了ImpScore的有效性，该研究将ImpScore的评估结果与人类在分布外数据上的评价进行了比较，证明了其准确性以及与人类判断的强相关性。此外，我们将ImpScore应用于仇恨言论检测数据集，展示了其实用性，并揭示了当前大型语言模型在理解高度隐含内容方面存在的显著局限性。