Understanding how helpful a visualization is from experimental results is difficult because the observed performance is confounded with aspects of the study design, such as how useful the information that is visualized is for the task. We develop a rational agent framework for designing and interpreting visualization experiments. Our framework conceives two experiments with the same setup: one with behavioral agents (human subjects), the other one with a hypothetical rational agent. A visualization is evaluated by comparing the expected performance of behavioral agents to that of rational agent under different assumptions. Using recent visualization decision studies from the literature, we demonstrate how the framework can be used to pre-experimentally evaluate the experiment design by bounding the expected improvement in performance from having access to visualizations, and post-experimentally to deconfound errors of information extraction from errors of optimization, among other analyses.

翻译：从实验结果中理解可视化的有效性较为困难，因为观测到的性能与实验设计的若干因素相互混淆，例如可视化信息对任务的有用程度。我们提出了一种用于设计和解释可视化实验的理性智能体框架。该框架设计了两组实验环境相同的实验：一组使用行为智能体（人类被试），另一组使用假设的理性智能体。通过比较行为智能体在不同假设下与理性智能体的期望性能，对可视化进行评估。我们利用文献中近期可视化决策研究，展示了该框架如何在实验前通过界定使用可视化后性能改进的期望上限来评估实验设计，并在实验后通过将信息提取误差与优化误差分离等方法进行去混淆分析。