Offline policy learning aims to discover decision-making policies from previously-collected datasets without additional online interactions with the environment. As the training dataset is fixed, its quality becomes a crucial determining factor in the performance of the learned policy. This paper studies a dataset characteristic that we refer to as multi-behavior, indicating that the dataset is collected using multiple policies that exhibit distinct behaviors. In contrast, a uni-behavior dataset would be collected solely using one policy. We observed that policies learned from a uni-behavior dataset typically outperform those learned from multi-behavior datasets, despite the uni-behavior dataset having fewer examples and less diversity. Therefore, we propose a behavior-aware deep clustering approach that partitions multi-behavior datasets into several uni-behavior subsets, thereby benefiting downstream policy learning. Our approach is flexible and effective; it can adaptively estimate the number of clusters while demonstrating high clustering accuracy, achieving an average Adjusted Rand Index of 0.987 across various continuous control task datasets. Finally, we present improved policy learning examples using dataset clustering and discuss several potential scenarios where our approach might benefit the offline policy learning community.

翻译：离线策略学习旨在从先前收集的数据集中发现决策策略，而无需与环境进行额外的在线交互。由于训练数据集是固定的，其质量成为所学策略性能的关键决定因素。本文研究了一种我们称之为多行为的数据集特征，表示该数据集是通过多种具有不同行为的策略收集的。相比之下，单行为数据集则仅使用一种策略收集。我们观察到，从单行为数据集学习的策略通常优于从多行为数据集学习的策略，尽管单行为数据集的样本更少且多样性较低。因此，我们提出了一种行为感知的深度聚类方法，将多行为数据集划分为多个单行为子集，从而有益于后续的策略学习。我们的方法灵活且有效：它能够自适应地估计聚类数量，同时展示出高聚类精度，在各种连续控制任务数据集中平均调整兰德指数达到0.987。最后，我们通过数据集聚类呈现了改进的策略学习实例，并讨论了我们的方法可能有益于离线策略学习社区的几种潜在场景。