Understanding how helpful a visualization is from experimental results is difficult because the observed performance is confounded with aspects of the study design, such as how useful the information that is visualized is for the task. We develop a rational agent framework for designing and interpreting visualization experiments. Our framework conceives two experiments with the same setup: one with behavioral agents (human subjects), and the other one with a hypothetical rational agent. A visualization is evaluated by comparing the expected performance of behavioral agents to that of a rational agent under different assumptions. Using recent visualization decision studies from the literature, we demonstrate how the framework can be used to pre-experimentally evaluate the experiment design by bounding the expected improvement in performance from having access to visualizations, and post-experimentally to deconfound errors of information extraction from errors of optimization, among other analyses.

翻译：从实验结果理解可视化的助益程度颇具难度，因为观测到的表现与研究设计中的其他因素（如所可视化的信息对任务的实用性）相互混杂。我们开发了一个理性智能体框架，用于设计和解读可视化实验。该框架构想两个具有相同设置条件的实验：一个采用行为智能体（人类受试者），另一个采用假设的理性智能体。通过比较行为智能体在不同假设条件下的预期表现与理性智能体的表现，对可视化进行评估。利用文献中近期可视化决策研究，我们展示了该框架如何通过约束可视化信息获取带来的预期性能改进上限，在实验前评估实验设计；并在实验后分离信息提取误差与优化误差及其他分析。