To understand how well a proposed augmented reality (AR) solution works, existing papers often conducted tailored and isolated evaluations for specific AR tasks, e.g., depth or lighting estimation, and compared them to easy-to-setup baselines, either using datasets or resorting to time-consuming data capturing. Conceptually simple, it can be extremely difficult to evaluate an AR system fairly and in scale to understand its real-world performance. The difficulties arise for three key reasons: lack of control of the physical environment, the time-consuming data capturing, and the difficulties to reproduce baseline results. This paper presents our design of an AR experimentation platform, ExpAR, aiming to provide scalable and controllable AR experimentation. ExpAR is envisioned to operate as a standalone deployment or a federated platform; in the latter case, AR researchers can contribute physical resources, including scene setup and capturing devices, and allow others to time share these resources. Our design centers around the generic sensing-understanding-rendering pipeline and is driven by the evaluation limitations observed in recent AR systems papers. We demonstrate the feasibility of this vision with a preliminary prototype and our preliminary evaluations suggest the importance of further investigating different device capabilities to stream in 30 FPS. The ExpAR project site can be found at https://cake.wpi.edu/expar.

翻译：为评估所提出的增强现实（AR）解决方案的实际效果，现有论文通常针对特定AR任务（如深度估计或光照估计）开展定制化且孤立的评估，并与易于搭建的基线方法进行比较，或使用数据集，或依赖耗时的数据采集流程。尽管概念上简单，但要在规模上公平地评估AR系统以理解其真实性能却极为困难。困难源于三个关键原因：物理环境缺乏可控性、数据采集耗时、以及基线结果难以复现。本文提出AR实验平台ExpAR的设计，旨在提供可扩展且可控的AR实验能力。ExpAR被构想为可独立部署或作为联邦化平台运行；在联邦模式下，AR研究人员可贡献物理资源（包括场景搭建与采集设备），并允许他人分时共享这些资源。我们的设计围绕通用的"感知-理解-渲染"流水线展开，并受到近期AR系统论文中评估局限性的启发。通过初步原型验证了这一愿景的可行性，初步评估表明进一步探索不同设备能力以实现30 FPS实时流传输至关重要。ExpAR项目网站详见https://cake.wpi.edu/expar。