We consider estimation and inference with data collected from episodic reinforcement learning (RL) algorithms; i.e. adaptive experimentation algorithms that at each period (aka episode) interact multiple times in a sequential manner with a single treated unit. Our goal is to be able to evaluate counterfactual adaptive policies after data collection and to estimate structural parameters such as dynamic treatment effects, which can be used for credit assignment (e.g. what was the effect of the first period action on the final outcome). Such parameters of interest can be framed as solutions to moment equations, but not minimizers of a population loss function, leading to Z-estimation approaches in the case of static data. However, such estimators fail to be asymptotically normal in the case of adaptive data collection. We propose a re-weighted Z-estimation approach with carefully designed adaptive weights to stabilize the episode-varying estimation variance, which results from the nonstationary policy that typical episodic RL algorithms invoke. We identify proper weighting schemes to restore the consistency and asymptotic normality of the re-weighted Z-estimators for target parameters, which allows for hypothesis testing and constructing reliable confidence regions for target parameters of interest. Primary applications include dynamic treatment effect estimation and dynamic off-policy evaluation.

翻译：我们考虑从经验性强化学习算法收集的数据中进行估计与推断的问题；即自适应性实验算法，该算法在每个周期（即剧集）中与单个处理单元以序列方式多次交互。我们的目标是在数据收集后能够评估反事实自适应策略，并估计动态处理效应等结构参数，这些参数可用于信用分配（例如，第一周期行动对最终结果的影响）。此类感兴趣的参数可以表述为矩方程的解，但并非总体损失函数的最小化者，这导致了在静态数据情况下的Z估计方法。然而，在自适应数据收集的情况下，此类估计量无法渐近正态。我们提出了一种重新加权的Z估计方法，该方法使用精心设计的自适应权重来稳定由典型经验性RL算法引发的非平稳策略导致的剧集变化估计方差。我们确定了适当的加权方案，以恢复目标参数的重加权Z估计量的一致性和渐近正态性，从而允许对感兴趣的目标参数进行假设检验并构建可靠的置信区域。主要应用包括动态处理效应估计和动态离线策略评估。