Adaptive bitrate (ABR) algorithms are used to adapt the video bitrate based on the network conditions to improve the overall video quality of experience (QoE). Recently, reinforcement learning (RL) and asynchronous advantage actor-critic (A3C) methods have been used to generate adaptive bit rate algorithms and they have been shown to improve the overall QoE as compared to fixed rule ABR algorithms. However, a common issue in the A3C methods is the lag between behaviour policy and target policy. As a result, the behaviour and the target policies are no longer synchronized which results in suboptimal updates. In this work, we present ALISA: An Actor-Learner Architecture with Importance Sampling for efficient learning in ABR algorithms. ALISA incorporates importance sampling weights to give more weightage to relevant experience to address the lag issues with the existing A3C methods. We present the design and implementation of ALISA, and compare its performance to state-of-the-art video rate adaptation algorithms including vanilla A3C implemented in the Pensieve framework and other fixed-rule schedulers like BB, BOLA, and RB. Our results show that ALISA improves average QoE by up to 25%-48% higher average QoE than Pensieve, and even more when compared to fixed-rule schedulers.

翻译：自适应比特率（ABR）算法根据网络条件调整视频码率，以提升整体视频体验质量（QoE）。近年来，强化学习（RL）和异步优势演员-评论家（A3C）方法被用于生成自适应比特率算法，并已被证明相比固定规则ABR算法能改善整体QoE。然而，A3C方法中的一个常见问题是行为策略与目标策略之间存在滞后。这导致行为策略和目标策略不再同步，从而产生次优更新。在本工作中，我们提出ALISA：一种利用重要性采样的演员-学习者架构，用于ABR算法的高效学习。ALISA引入重要性采样权重，对相关经验赋予更高权重，以解决现有A3C方法中的滞后问题。我们介绍了ALISA的设计与实现，并将其性能与包括Pensieve框架中实现的原始A3C以及BB、BOLA、RB等固定规则调度器在内的最先进视频码率自适应算法进行比较。结果表明，ALISA的平均QoE比Pensieve提高25%-48%，相较于固定规则调度器提升更为显著。