Numerous offline and model-based reinforcement learning systems incorporate world models to emulate the inherent environments. A world model is particularly important in scenarios where direct interactions with the real environment is costly, dangerous, or impractical. The efficacy and interpretability of such world models are notably contingent upon the quality of the underlying training data. In this context, we introduce Action Shapley as an agnostic metric for the judicious and unbiased selection of training data. To facilitate the computation of Action Shapley, we present a randomized dynamic algorithm specifically designed to mitigate the exponential complexity inherent in traditional Shapley value computations. Through empirical validation across five data-constrained real-world case studies, the algorithm demonstrates a computational efficiency improvement exceeding 80\% in comparison to conventional exponential time computations. Furthermore, our Action Shapley-based training data selection policy consistently outperforms ad-hoc training data selection.

翻译：众多离线和基于模型的强化学习系统采用世界模型来模拟内在环境。在与真实环境直接交互成本高昂、危险或不切实际的情况下，世界模型尤为重要。此类世界模型的效能与可解释性显著依赖于其底层训练数据的质量。在此背景下，我们提出动作沙普利值作为一种与模型无关的度量标准，用于训练数据的合理且无偏选择。为促进动作沙普利值的计算，我们提出一种随机动态算法，专门设计用于缓解传统沙普利值计算中固有的指数级复杂度。通过在五个数据受限的真实世界案例研究中进行实证验证，该算法相较于传统的指数时间计算展现出超过80%的计算效率提升。此外，我们基于动作沙普利值的训练数据选择策略持续优于临时性的训练数据选择方法。