Understanding how helpful a visualization is from experimental results is difficult because the observed performance is confounded with aspects of the study design, such as how useful the information that is visualized is for the task. We develop a rational agent framework for designing and interpreting visualization experiments. Our framework conceives two experiments with the same setup: one with behavioral agents (human subjects), and the other one with a hypothetical rational agent. A visualization is evaluated by comparing the expected performance of behavioral agents to that of a rational agent under different assumptions. Using recent visualization decision studies from the literature, we demonstrate how the framework can be used to pre-experimentally evaluate the experiment design by bounding the expected improvement in performance from having access to visualizations, and post-experimentally to deconfound errors of information extraction from errors of optimization, among other analyses.

翻译：从实验结果理解可视化的有用性十分困难，因为观测到的性能与研究设计的诸多方面（如任务中可视化信息的实用价值）混杂在一起。我们开发了一个用于设计和解释可视化实验的理性主体框架。该框架设计了两个具有相同设置的实验：一个使用行为主体（人类受试者），另一个使用假设的理性主体。通过比较行为主体在不同假设下的预期表现与理性主体的表现来评估可视化效果。利用文献中近期的可视化决策研究，我们演示了该框架如何通过约束从可视化获取信息带来的预期性能改进空间来对实验设计进行事前评估，以及如何在事后分析中分离信息提取误差与优化误差等问题。