While reinforcement learning (RL) algorithms have been successfully applied across numerous sequential decision-making problems, their generalization to unforeseen testing environments remains a significant concern. In this paper, we study the problem of out-of-distribution (OOD) detection in RL, which focuses on identifying situations at test time that RL agents have not encountered in their training environments. We first propose a clarification of terminology for OOD detection in RL, which aligns it with the literature from other machine learning domains. We then present new benchmark scenarios for OOD detection, which introduce anomalies with temporal autocorrelation into different components of the agent-environment loop. We argue that such scenarios have been understudied in the current literature, despite their relevance to real-world situations. Confirming our theoretical predictions, our experimental results suggest that state-of-the-art OOD detectors are not able to identify such anomalies. To address this problem, we propose a novel method for OOD detection, which we call DEXTER (Detection via Extraction of Time Series Representations). By treating environment observations as time series data, DEXTER extracts salient time series features, and then leverages an ensemble of isolation forest algorithms to detect anomalies. We find that DEXTER can reliably identify anomalies across benchmark scenarios, exhibiting superior performance compared to both state-of-the-art OOD detectors and high-dimensional changepoint detectors adopted from statistics.

翻译：尽管强化学习算法已成功应用于众多序贯决策问题，但其在未见测试环境中的泛化能力仍是一个重要挑战。本文研究强化学习中的分布外检测问题，该问题聚焦于识别测试时强化学习智能体在其训练环境中未曾遭遇的情境。我们首先提出强化学习分布外检测术语的澄清方案，使其与机器学习其他领域的文献保持一致。随后针对分布外检测提出新的基准场景，这些场景将具有时间自相关性的异常现象引入智能体-环境循环的不同组件中。我们认为，尽管此类场景与现实情境高度相关，却在当前文献中鲜有研究。实验证实了我们的理论预测：现有最先进的分布外检测器无法识别此类异常。为解决该问题，我们提出一种名为DEXTER（基于时间序列表征提取的检测方法）的新型分布外检测方法。通过将环境观测视为时间序列数据，DEXTER提取显著的时间序列特征，并利用孤立森林算法的集成来检测异常。实验表明，DEXTER能够在基准场景中可靠识别异常，其性能优于当前最先进的分布外检测器以及统计学领域引入的高维变点检测方法。