Online crowdsourcing platforms have made it increasingly easy to perform evaluations of algorithm outputs with survey questions like "which image is better, A or B?", leading to their proliferation in vision and graphics research papers. Results of these studies are often used as quantitative evidence in support of a paper's contributions. On the one hand we argue that, when conducted hastily as an afterthought, such studies lead to an increase of uninformative, and, potentially, misleading conclusions. On the other hand, in these same communities, user research is underutilized in driving project direction and forecasting user needs and reception. We call for increased attention to both the design and reporting of user studies in computer vision and graphics papers towards (1) improved replicability and (2) improved project direction. Together with this call, we offer an overview of methodologies from user experience research (UXR), human-computer interaction (HCI), and applied perception to increase exposure to the available methodologies and best practices. We discuss foundational user research methods (e.g., needfinding) that are presently underutilized in computer vision and graphics research, but can provide valuable project direction. We provide further pointers to the literature for readers interested in exploring other UXR methodologies. Finally, we describe broader open issues and recommendations for the research community.

翻译：在线众包平台使得通过"图A和图B哪个更好？"等调查问题对算法输出进行评估日益便捷，这促使此类研究在视觉与图形学论文中大量涌现。这些研究结果常被用作支撑论文贡献的定量证据。一方面我们认为，若将此类研究仓促视为事后补充，将导致无信息量甚至具有潜在误导性的结论增多。另一方面，在这些学术社群中，用户研究在驱动项目方向、预测用户需求与接受度方面尚未得到充分利用。我们呼吁计算机视觉与图形学论文关注用户研究的设计与报告，以实现：（1）提升可复现性；（2）优化项目方向。伴随这一倡议，我们系统梳理了用户体验研究、人机交互与应用感知领域的方法论，以增进学界对现有方法论与最佳实践的认知。我们探讨了当前在计算机视觉与图形学研究中未被充分运用、但能为项目方向提供宝贵指引的基础用户研究方法（如需求发现）。为有意探索其他用户体验研究方法的读者提供文献指引。最后，我们阐述更广泛的开放议题并向研究界提出建议。