Anomaly detection in decision-making sequences is a challenging problem due to the complexity of normality representation learning and the sequential nature of the task. Most existing methods based on Reinforcement Learning (RL) are difficult to implement in the real world due to unrealistic assumptions, such as having access to environment dynamics, reward signals, and online interactions with the environment. To address these limitations, we propose an unsupervised method named Offline Imitation Learning based Anomaly Detection (OIL-AD), which detects anomalies in decision-making sequences using two extracted behaviour features: action optimality and sequential association. Our offline learning model is an adaptation of behavioural cloning with a transformer policy network, where we modify the training process to learn a Q function and a state value function from normal trajectories. We propose that the Q function and the state value function can provide sufficient information about agents' behavioural data, from which we derive two features for anomaly detection. The intuition behind our method is that the action optimality feature derived from the Q function can differentiate the optimal action from others at each local state, and the sequential association feature derived from the state value function has the potential to maintain the temporal correlations between decisions (state-action pairs). Our experiments show that OIL-AD can achieve outstanding online anomaly detection performance with up to 34.8% improvement in F1 score over comparable baselines.

翻译：决策序列中的异常检测因常态表征学习的复杂性及任务的序列特性而极具挑战性。现有基于强化学习的方法大多依赖环境动力学、奖励信号及在线交互等不切实际的假设，导致其难以在真实场景中部署。为解决上述局限，我们提出了一种名为"基于离线模仿学习的异常检测"的无监督方法——该方法通过提取两种行为特征（动作最优性与序列关联性）来检测决策序列中的异常。我们的离线学习模型基于行为克隆与Transformer策略网络进行适配改造，通过修改训练流程从正常轨迹中学习Q函数与状态价值函数。理论分析表明，Q函数与状态价值函数能够充分表征智能体行为数据中的关键信息，据此我们可推导出两种用于异常检测的特征。其核心思想在于：由Q函数推导的动作最优性特征可在局部状态下区分最优动作与其他动作，而由状态价值函数推导的序列关联性特征则能有效保持决策过程（状态-动作对）的时间相关性。实验结果表明，OIL-AD在在线异常检测任务中表现出色，F1分数相较同类基线方法最高提升34.8%。