The recent development of reinforcement learning (RL) has boosted the adoption of online RL for wireless radio resource management (RRM). However, online RL algorithms require direct interactions with the environment, which may be undesirable given the potential performance loss due to the unavoidable exploration in RL. In this work, we first investigate the use of \emph{offline} RL algorithms in solving the RRM problem. We evaluate several state-of-the-art offline RL algorithms, including behavior constrained Q-learning (BCQ), conservative Q-learning (CQL), and implicit Q-learning (IQL), for a specific RRM problem that aims at maximizing a linear combination {of sum and} 5-percentile rates via user scheduling. We observe that the performance of offline RL for the RRM problem depends critically on the behavior policy used for data collection, and further propose a novel offline RL solution that leverages heterogeneous datasets collected by different behavior policies. We show that with a proper mixture of the datasets, offline RL can produce a near-optimal RL policy even when all involved behavior policies are highly suboptimal.

翻译：强化学习（RL）的最新发展推动了在线RL在无线资源管理（RRM）中的应用。然而，在线RL算法需要与环境直接交互，由于RL中不可避免的探索可能导致性能损失，这可能是不可取的。本文首先研究将离线RL算法用于解决RRM问题。我们针对一个旨在通过用户调度最大化{总和与}5百分位速率的线性组合的特定RRM问题，评估了包括行为约束Q学习（BCQ）、保守Q学习（CQL）和隐式Q学习（IQL）在内的几种先进离线RL算法。我们观察到，离线RL在RRM问题上的性能关键取决于用于数据收集的行为策略，并进一步提出了一种新颖的离线RL解决方案，该方案利用不同行为策略收集的异构数据集。我们证明，通过适当混合数据集，即使所有涉及的行为策略都高度次优，离线RL也能产生接近最优的RL策略。