Estimating optimal dynamic policies from offline data is a fundamental problem in dynamic decision making. In the context of causal inference, the problem is known as estimating the optimal dynamic treatment regime. Even though there exists a plethora of methods for estimation, constructing confidence intervals for the value of the optimal regime and structural parameters associated with it is inherently harder, as it involves non-linear and non-differentiable functionals of un-known quantities that need to be estimated. Prior work resorted to sub-sample approaches that can deteriorate the quality of the estimate. We show that a simple soft-max approximation to the optimal treatment regime, for an appropriately fast growing temperature parameter, can achieve valid inference on the truly optimal regime. We illustrate our result for a two-period optimal dynamic regime, though our approach should directly extend to the finite horizon case. Our work combines techniques from semi-parametric inference and $g$-estimation, together with an appropriate triangular array central limit theorem, as well as a novel analysis of the asymptotic influence and asymptotic bias of softmax approximations.

翻译：从离线数据估计最优动态策略是动态决策中的一个基本问题。在因果推断的背景下，该问题被称为估计最优动态处理方案。尽管存在大量估计方法，但为最优策略的价值及其相关结构参数构建置信区间本质上更为困难，因为这涉及需要估计的未知量的非线性且不可微泛函。以往研究采用子样本方法，这可能会降低估计质量。我们证明，对于适当快速增长的温度参数，对最优处理方案进行简单的Softmax近似可以针对真正的最优策略实现有效的推断。我们以两期最优动态方案为例展示了结果，尽管我们的方法应直接适用于有限时域情形。本研究结合了半参数推断和g-估计技术，并应用了适当的三角阵列中心极限定理，以及对Softmax近似渐近影响和渐近偏差的新颖分析。