This work explores the utility of explicit structure for representation learning in NLP by developing StrAE -- an autoencoding framework that faithfully leverages sentence structure to learn multi-level node embeddings in an unsupervised fashion. We use StrAE to train models across different types of sentential structure and objectives, including a novel contrastive loss over structure, and evaluate the learnt embeddings on a series of both intrinsic and extrinsic tasks. Our experiments indicate that leveraging explicit structure through StrAE leads to improved embeddings over prior work, and that our novel contrastive objective over structure outperforms the standard cross-entropy objective. Moreover, in contrast to findings from prior work that weakly leverages structure, we find that being completely faithful to structure does enable disambiguation between types of structure based on the corresponding model's performance. As further evidence of StrAE's utility, we develop a simple proof-of-concept approach to simultaneously induce structure while learning embeddings, rather than being given structure, and find that performance is comparable to that of the best-performing models where structure is given. Finally, we contextualise these results by comparing StrAE against standard unstructured baselines learnt in similar settings, and show that faithfully leveraging explicit structure can be beneficial in lexical and sentence-level semantics.

翻译：本工作探索了显式结构在自然语言处理表示学习中的效用，通过开发StrAE——一种自编码框架，能够忠实地利用句子结构以无监督方式学习多层级节点嵌入。我们使用StrAE在不同类型的句子结构和目标下训练模型，包括一种新颖的基于结构的对比损失，并在系列内在和外在任务中评估学习到的嵌入。实验表明，通过StrAE利用显式结构可获得优于先前工作的嵌入，且我们提出的结构对比目标优于标准交叉熵目标。此外，与弱利用结构的先前研究结论不同，我们发现完全忠实于结构确实能根据对应模型的性能区分结构类型。作为StrAE效用的进一步证据，我们开发了一种简单的概念验证方法，在学习嵌入的同时诱导结构而非预置结构，发现其性能与给定结构的最佳模型相当。最后，通过将StrAE与相似设置下学习的标准无结构基线进行比较，我们论证了基于结构的嵌入结果，并表明忠实地利用显式结构在词汇和句子级语义中具有优势。