Establishing whether language models can use contextual information in a human-plausible way is important to ensure their trustworthiness in real-world settings. However, the questions of when and which parts of the context affect model generations are typically tackled separately, with current plausibility evaluations being practically limited to a handful of artificial benchmarks. To address this, we introduce Plausibility Evaluation of Context Reliance (PECoRe), an end-to-end interpretability framework designed to quantify context usage in language models' generations. Our approach leverages model internals to (i) contrastively identify context-sensitive target tokens in generated texts and (ii) link them to contextual cues justifying their prediction. We use \pecore to quantify the plausibility of context-aware machine translation models, comparing model rationales with human annotations across several discourse-level phenomena. Finally, we apply our method to unannotated model translations to identify context-mediated predictions and highlight instances of (im)plausible context usage throughout generation.

翻译：确定语言模型能否以类人方式使用上下文信息，对于确保其在现实场景中的可信度至关重要。然而，关于上下文何时以及哪些部分影响模型生成的问题通常被分开处理，而当前的可信度评估实际上仅限于少数人工基准测试。为解决这一问题，我们提出了上下文依赖可信度评估框架（PECoRe），这是一个端到端的可解释性框架，旨在量化语言模型生成文本中的上下文使用情况。我们的方法利用模型内部机制来（i）对比性地识别生成文本中对上下文敏感的标记，以及（ii）将这些标记与证明其预测合理性的上下文线索关联起来。我们使用PECoRe量化上下文感知机器翻译模型的可信度，通过跨多个话语级现象比较模型解释与人工标注。最后，我们将该方法应用于未标注的模型翻译结果，以识别由上下文驱动的预测，并突出生成过程中上下文使用的（不）可信实例。