Understanding how helpful a visualization is from experimental results is difficult because the observed performance is confounded with aspects of the study design, such as how useful the information that is visualized is for the task. We develop a rational agent framework for designing and interpreting visualization experiments. Our framework conceives two experiments with the same setup: one with behavioral agents (human subjects), the other one with a hypothetical rational agent. A visualization is evaluated by comparing the expected performance of behavioral agents to that of rational agent under different assumptions. Using recent visualization decision studies from the literature, we demonstrate how the framework can be used to pre-experimentally evaluate the experiment design by bounding the expected improvement in performance from having access to visualizations, and post-experimentally to deconfound errors of information extraction from errors of optimization, among other analyses.

翻译：从实验结果中理解可视化对任务的有用程度存在困难，因为观察到的表现会与研究设计中的其他因素（例如可视化信息对任务的实际效用）产生混淆。我们提出一个理性主体框架，用于设计和解释可视化实验。该框架基于相同实验设置设计两类实验：一类使用行为主体（人类被试），另一类使用假设的理性主体。通过比较行为主体在不同假设下的期望表现与理性主体的表现，来评估可视化效果。我们利用文献中近期可视化决策研究，展示了该框架可如何用于实验前评估：通过界定因获得可视化而产生的期望性能改进上限来检验实验设计；以及实验后分析：将信息提取错误与优化错误进行解耦等分析。