In this thesis, we develop methods to enhance the interpretability of recent representation learning techniques in natural language processing (NLP) while accounting for the unavailability of annotated data. We choose to leverage Variational Autoencoders (VAEs) due to their efficiency in relating observations to latent generative factors and their effectiveness in data-efficient learning and interpretable representation learning. As a first contribution, we identify and remove unnecessary components in the functioning scheme of semi-supervised VAEs making them faster, smaller and easier to design. Our second and main contribution is to use VAEs and Transformers to build two models with inductive bias to separate information in latent representations into understandable concepts without annotated data. The first model, Attention-Driven VAE (ADVAE), is able to separately represent and control information about syntactic roles in sentences. The second model, QKVAE, uses separate latent variables to form keys and values for its Transformer decoder and is able to separate syntactic and semantic information in its neural representations. In transfer experiments, QKVAE has competitive performance compared to supervised models and equivalent performance to a supervised model using 50K annotated samples. Additionally, QKVAE displays improved syntactic role disentanglement capabilities compared to ADVAE. Overall, we demonstrate that it is possible to enhance the interpretability of state-of-the-art deep learning architectures for language modeling with unannotated data in situations where text data is abundant but annotations are scarce.

翻译：本文针对自然语言处理（NLP）中缺乏标注数据的情况，提出增强当前表征学习技术可解释性的方法。我们选择利用变分自编码器（VAE），因其在关联观测数据与潜在生成因子方面的高效性，以及在数据高效学习和可解释表征学习中的有效性。作为第一项贡献，我们识别并移除了半监督VAE运行机制中的非必要组件，使其更快速、轻量且易于设计。第二项也是主要贡献在于，我们结合VAE与Transformer构建了两个具备归纳偏置的模型，无需标注数据即可将潜在表征中的信息分离为可理解的概念。第一个模型——注意力驱动VAE（ADVAE）——能够独立表征并控制句子中句法角色的信息。第二个模型QKVAE通过分离的潜在变量为其Transformer解码器生成键与值，成功实现神经表征中句法与语义信息的分离。迁移实验中，QKVAE性能与监督模型相当，甚至在使用5万标注样本时与监督模型表现持平。此外，相较于ADVAE，QKVAE展现出更优的句法角色解耦能力。总体而言，我们证明了在文本数据丰富但标注稀缺的场景下，通过无标注数据增强语言建模领域最先进深度学习架构的可解释性是可行的。