We consider the problem of finding the optimal value of n in the n-step temporal difference (TD) algorithm. We find the optimal n by resorting to the model-free optimization technique of simultaneous perturbation stochastic approximation (SPSA). We adopt a one-simulation SPSA procedure that is originally for continuous optimization to the discrete optimization framework but incorporates a cyclic perturbation sequence. We prove the convergence of our proposed algorithm, SDPSA, and show that it finds the optimal value of n in n-step TD. Through experiments, we show that the optimal value of n is achieved with SDPSA for any arbitrary initial value of the same.

翻译：我们考虑在n步时间差分（TD）算法中寻找最优n值的问题。通过采用无模型优化技术——同时扰动随机逼近（SPSA），我们求得了最优的n。我们将原本用于连续优化的单次仿真SPSA过程适配到离散优化框架中，并引入循环扰动序列。我们证明了所提出的算法SDPSA的收敛性，并表明该算法能够在n步TD中找到最优n值。实验结果表明，对于任意给定的初始n值，SDPSA均能获得最优n值。