Can the causal effects estimated in experiment be generalized to real-world scenarios? This question lies at the heart of social science studies. External validity primarily assesses whether experimental effects persist across different settings, implicitly presuming the experiment's ecological validity-that is, the consistency of experimental effects with their real-life counterparts. However, we argue that this presumed consistency may not always hold, especially in experiments involving multidimensional decision processes, such as conjoint experiments. We introduce a formal model to elucidate how attention and salience effects lead to three types of inconsistencies between experimental findings and real-world phenomena: amplified effect magnitude, effect sign reversal, and effect importance reversal. We derive testable hypotheses from each theoretical outcome and test these hypotheses using data from various existing conjoint experiments and our own experiments. Drawing on our theoretical framework, we propose several recommendations for experimental design aimed at enhancing the generalizability of survey experiment findings.

翻译：在实验中估计的因果效应能否推广到现实场景？这一问题处于社会科学研究的核心。外部效度主要评估实验效应是否在不同情境中持续存在，这隐含地假定了实验的生态效度——即实验效应与其现实对应物的一致性。然而，我们认为这种假定的一致性可能并非总是成立，尤其是在涉及多维决策过程的实验中，例如联合实验。我们引入了一个形式化模型，用以阐明注意力和显著性效应如何导致实验发现与现实现象之间的三类不一致：效应幅度放大、效应符号反转以及效应重要性反转。我们为每个理论结果推导出可检验的假设，并利用来自现有多个联合实验的数据以及我们自己的实验数据对这些假设进行检验。基于我们的理论框架，我们提出若干旨在提升调查实验结果泛化性的实验设计建议。