Offline reinforcement learning (RL) presents a promising approach for learning reinforced policies from offline datasets without the need for costly or unsafe interactions with the environment. However, datasets collected by humans in real-world environments are often noisy and may even be maliciously corrupted, which can significantly degrade the performance of offline RL. In this work, we first investigate the performance of current offline RL algorithms under comprehensive data corruption, including states, actions, rewards, and dynamics. Our extensive experiments reveal that implicit Q-learning (IQL) demonstrates remarkable resilience to data corruption among various offline RL algorithms. Furthermore, we conduct both empirical and theoretical analyses to understand IQL's robust performance, identifying its supervised policy learning scheme as the key factor. Despite its relative robustness, IQL still suffers from heavy-tail targets of Q functions under dynamics corruption. To tackle this challenge, we draw inspiration from robust statistics to employ the Huber loss to handle the heavy-tailedness and utilize quantile estimators to balance penalization for corrupted data and learning stability. By incorporating these simple yet effective modifications into IQL, we propose a more robust offline RL approach named Robust IQL (RIQL). Extensive experiments demonstrate that RIQL exhibits highly robust performance when subjected to diverse data corruption scenarios.

翻译：离线强化学习（Offline RL）提供了一种有前景的方法，能够从离线数据集中学习强化策略，而无需进行昂贵或危险的环境交互。然而，现实环境中由人类收集的数据集往往包含噪声，甚至可能遭受恶意破坏，这会显著降低离线强化学习的性能。本文首先研究了当前离线强化学习算法在全面数据损坏（包括状态、动作、奖励和动态）下的性能表现。广泛实验表明，隐式Q学习（IQL）在众多离线强化学习算法中对数据损坏展现出显著的鲁棒性。此外，我们通过经验与理论分析揭示了IQL鲁棒性能的成因，将其监督式策略学习方案识别为关键因素。尽管IQL相对鲁棒，但在动态损坏条件下仍面临Q函数重尾目标的问题。为应对该挑战，我们借鉴鲁棒统计思想，采用Huber损失处理重尾性，并利用分位数估计器平衡对损坏数据的惩罚与学习稳定性。通过将这些简单而有效的改进融入IQL，我们提出了一种更鲁棒的离线强化学习方法——鲁棒IQL（RIQL）。大量实验表明，RIQL在多种数据损坏场景下展现出高度鲁棒的性能。