Understanding how helpful a visualization is from experimental results is difficult because the observed performance is confounded with aspects of the study design, such as how useful the information that is visualized is for the task. We develop a rational agent framework for designing and interpreting visualization experiments. Our framework conceives two experiments with the same setup: one with behavioral agents (human subjects), and the other one with a hypothetical rational agent. A visualization is evaluated by comparing the expected performance of behavioral agents to that of a rational agent under different assumptions. Using recent visualization decision studies from the literature, we demonstrate how the framework can be used to pre-experimentally evaluate the experiment design by bounding the expected improvement in performance from having access to visualizations, and post-experimentally to deconfound errors of information extraction from errors of optimization, among other analyses.

翻译：从实验结果理解可视化辅助效果的难度在于，观察到的表现与实验设计中的其他因素（如任务所依赖信息的有用性）存在混杂效应。我们提出了一个理性代理框架用于设计和解读可视化实验。该框架在相同实验设置下并行开展两类实验：一类使用行为代理（人类受试者），另一类使用假设的理性代理。通过比较不同假设条件下行为代理与理性代理的预期表现来评估可视化效果。基于文献中最新可视化决策研究，我们展示了该框架如何通过约束可视化带来的预期性能提升边界来进行实验前评估设计，以及在实验后解耦信息提取误差与优化误差等分析方法。