We consider the problem of finding the optimal value of n in the n-step temporal difference (TD) learning algorithm. We find the optimal n by resorting to a model-free optimization technique involving a one-simulation simultaneous perturbation stochastic approximation (SPSA) based procedure that we adopt to the discrete optimization setting by using a random projection approach. We prove the convergence of our proposed algorithm, SDPSA, using a differential inclusions approach and show that it finds the optimal value of n in n-step TD. Through experiments, we show that the optimal value of n is achieved with SDPSA for arbitrary initial values.

翻译：我们研究在n步时序差分（TD）学习算法中寻找最优n值的问题。通过采用一种基于单次仿真的同步扰动随机逼近（SPSA）过程的无模型优化技术，并利用随机投影方法将其适配到离散优化场景，我们找到了最优n值。利用微分包含方法证明了所提出算法SDPSA的收敛性，并表明该算法能够找到n步TD中的最优n值。实验证明，SDPSA算法能够针对任意初始值获得最优n值。