We consider the problem of finding the optimal value of n in the n-step temporal difference (TD) algorithm. We find the optimal n by resorting to the model-free optimization technique of simultaneous perturbation stochastic approximation (SPSA). We adopt a one-simulation SPSA procedure that is originally for continuous optimization to the discrete optimization framework but incorporates a cyclic perturbation sequence. We prove the convergence of our proposed algorithm, SDPSA, and show that it finds the optimal value of n in n-step TD. Through experiments, we show that the optimal value of n is achieved with SDPSA for any arbitrary initial value of the same.

翻译：我们考虑在n步时序差分（TD）算法中寻找最优n值的问题。通过引入无模型优化技术——同时扰动随机逼近（SPSA），我们实现了最优n值的求解。我们将原本用于连续优化的单仿真SPSA过程适配至离散优化框架，并采用循环扰动序列。我们证明了所提出的算法SDPSA的收敛性，并表明该算法能够找到n步TD中的最优n值。实验结果表明，对于任意初始n值，SDPSA均能收敛至最优n值。