The neural architectures of language models are becoming increasingly complex, especially that of Transformers, based on the attention mechanism. Although their application to numerous natural language processing tasks has proven to be very fruitful, they continue to be models with little or no interpretability and explainability. One of the tasks for which they are best suited is the encoding of the contextual sense of words using contextualized embeddings. In this paper we propose a transparent, interpretable, and linguistically motivated strategy for encoding the contextual sense of words by modeling semantic compositionality. Particular attention is given to dependency relations and semantic notions such as selection preferences and paradigmatic classes. A partial implementation of the proposed model is carried out and compared with Transformer-based architectures for a given semantic task, namely the similarity calculation of word senses in context. The results obtained show that it is possible to be competitive with linguistically motivated models instead of using the black boxes underlying complex neural architectures.

翻译：语言模型的神经结构正变得越来越复杂，尤其是基于注意力机制的Transformer。尽管它们在众多自然语言处理任务中的应用已被证明非常有效，但这些模型仍然缺乏可解释性和可解释性。它们最擅长的任务之一是利用上下文嵌入来编码词语的上下文意义。本文提出了一种透明、可解释且具有语言学动机的策略，通过建模语义组合性来编码词语的上下文意义。特别关注依存关系以及语义概念（如选择偏好和聚合类）。我们对所提出的模型进行了部分实现，并与基于Transformer的结构在特定语义任务（即计算上下文中词义的相似度）上进行了比较。结果表明，使用具有语言学动机的模型而不是依赖复杂神经架构背后的黑箱，同样可以具备竞争力。