Experimental methods for estimating the impacts of text on human evaluation have been widely used in the social sciences. However, researchers in experimental settings are usually limited to testing a small number of pre-specified text treatments. While efforts to mine unstructured texts for features that causally affect outcomes have been ongoing in recent years, these models have primarily focused on the topics or specific words of text, which may not always be the mechanism of the effect. We connect these efforts with NLP interpretability techniques and present a method for flexibly discovering clusters of similar text phrases that are predictive of human reactions to texts using convolutional neural networks. When used in an experimental setting, this method can identify text treatments and their effects under certain assumptions. We apply the method to two datasets. The first enables direct validation of the model's ability to detect phrases known to cause the outcome. The second demonstrates its ability to flexibly discover text treatments with varying textual structures. In both cases, the model learns a greater variety of text treatments compared to benchmark methods, and these text features quantitatively meet or exceed the ability of benchmark methods to predict the outcome.

翻译：用于评估文本对人类评价影响的实验方法已在社会科学领域得到广泛应用。然而，实验环境中的研究者通常仅限于测试少量预先设定的文本处理条件。尽管近年来持续存在从非结构化文本中挖掘对结果产生因果影响特征的研究尝试，但这些模型主要关注文本主题或特定词汇，而这些可能并非总是产生影响的机制。我们将这些研究尝试与自然语言处理可解释性技术相结合，提出一种利用卷积神经网络灵活发现能够预测人类对文本反应的相似文本短语簇的方法。在实验环境中应用时，该方法能够在特定假设条件下识别文本处理条件及其效应。我们将该方法应用于两个数据集：第一个数据集可直接验证模型检测已知导致结果短语的能力；第二个数据集则展示其灵活发现具有不同文本结构的文本处理条件的能力。在两种情况下，相较于基准方法，该模型能够学习到更多样化的文本处理条件，且这些文本特征在预测结果方面的能力在定量分析中达到或超越了基准方法的水平。