For video or web services, it is crucial to measure user-perceived quality of experience (QoE) at scale under various video quality or page loading delays. However, fast QoE measurements remain challenging as they must elicit subjective assessment from human users. Previous work either (1) automates QoE measurements by letting crowdsourcing raters watch and rate QoE test videos or (2) dynamically prunes redundant QoE tests based on previously collected QoE measurements. Unfortunately, it is hard to combine both ideas because traditional crowdsourcing requires QoE test videos to be pre-determined before a crowdsourcing campaign begins. Thus, if researchers want to dynamically prune redundant test videos based on other test videos' QoE, they are forced to launch multiple crowdsourcing campaigns, causing extra overheads to re-calibrate or train raters every time. This paper presents VidPlat, the first open-source tool for fast and automated QoE measurements, by allowing dynamic pruning of QoE test videos within a single crowdsourcing task. VidPlat creates an indirect shim layer between researchers and the crowdsourcing platforms. It allows researchers to define a logic that dynamically determines which new test videos need more QoE ratings based on the latest QoE measurements, and it then redirects crowdsourcing raters to watch QoE test videos dynamically selected by this logic. Other than having fewer crowdsourcing campaigns, VidPlat also reduces the total number of QoE ratings by dynamically deciding when enough ratings are gathered for each test video. It is an open-source platform that future researchers can reuse and customize. We have used VidPlat in three projects (web loading, on-demand video, and online gaming). We show that VidPlat can reduce crowdsourcing cost by 31.8% - 46.0% and latency by 50.9% - 68.8%.

翻译：对于视频或网络服务而言，在不同视频质量或页面加载延迟条件下规模化测量用户感知的体验质量（QoE）至关重要。然而，快速QoE测量仍然具有挑战性，因为这必须引发人类用户的主观评估。以往的研究要么（1）通过让众包评分者观看并评价QoE测试视频来自动化QoE测量，要么（2）基于先前收集的QoE测量结果动态剪枝冗余的QoE测试。遗憾的是，这两种思路难以结合，因为传统众包要求QoE测试视频在众包活动开始前预先确定。因此，若研究者希望根据其他测试视频的QoE动态剪枝冗余测试视频，则需启动多个众包活动，导致每次需重新校准或训练评分者而产生额外开销。本文提出VidPlat——首个支持在单个众包任务中动态剪枝QoE测试视频的快速自动化QoE测量开源工具。VidPlat在研究者与众包平台之间构建了一个间接的垫片层。它允许研究者定义逻辑，根据最新QoE测量结果动态判断哪些新测试视频需要更多QoE评分，并引导众包评分者观看由此逻辑动态选择的QoE测试视频。除减少众包活动数量外，VidPlat还通过动态判定每个测试视频何时收集足够评分来降低QoE评分总量。这是一个未来研究者可复用与定制的开源平台。我们已在三个项目（网页加载、点播视频和在线游戏）中应用VidPlat。实验表明，VidPlat可降低31.8%-46.0%的众包成本，并减少50.9%-68.8%的延迟。