Adaptive design optimization (ADO) is a state-of-the-art technique for experimental design (Cavagnaro, Myung, Pitt, & Kujala, 2010). ADO dynamically identifies stimuli that, in expectation, yield the most information about a hypothetical construct of interest (e.g., parameters of a cognitive model). To calculate this expectation, ADO leverages the modeler's existing knowledge, specified in the form of a prior distribution. Informative priors align with the distribution of the focal construct in the participant population. This alignment is assumed by ADO's internal assessment of expected information gain. If the prior is instead misinformative, i.e., does not align with the participant population, ADO's estimates of expected information gain could be inaccurate. In many cases, the true distribution that characterizes the participant population is unknown, and experimenters rely on heuristics in their choice of prior and without an understanding of how this choice affects ADO's behavior. Our work introduces a mathematical framework that facilitates investigation of the consequences of the choice of prior distribution on the efficiency of experiments designed using ADO. Through theoretical and empirical results, we show that, in the context of prior misinformation, measures of expected information gain are distinct from the correctness of the corresponding inference. Through a series of simulation experiments, we show that, in the case of parameter estimation, ADO nevertheless outperforms other design methods. Conversely, in the case of model selection, misinformative priors can lead inference to favor the wrong model, and rather than mitigating this pitfall, ADO exacerbates it.

翻译：自适应设计优化（ADO）是实验设计领域的一项前沿技术（Cavagnaro, Myung, Pitt, & Kujala, 2010）。ADO能够动态识别预期能最大程度提供关于目标假设构念（如认知模型参数）信息的刺激。为计算这一预期，ADO利用建模者以先验分布形式表达的现有知识。信息性先验与参与者群体中焦点构念的分布相一致，ADO内部对预期信息增益的评估假定这种一致性成立。若先验具有误导性（即与参与者群体分布不一致），则ADO对预期信息增益的估计可能不准确。在许多情况下，刻画参与者群体真实分布的特征未知，实验者依赖启发式方法选择先验，且不了解这种选择如何影响ADO的行为。本研究引入一个数学框架，便于探究先验分布选择对使用ADO设计的实验效率的影响。通过理论与实证结果，我们证明在先验误导的情况下，预期信息增益的度量与相应推断的正确性存在差异。通过一系列仿真实验，我们表明在参数估计场景中，ADO仍优于其他设计方法。相反，在模型选择场景中，误导性先验可能导致推断偏向错误模型，而ADO不仅未能缓解这一缺陷，反而加剧了该问题。