We consider stochastic optimization problems where data is drawn from a Markov chain. Existing methods for this setting crucially rely on knowing the mixing time of the chain, which in real-world applications is usually unknown. We propose the first optimization method that does not require the knowledge of the mixing time, yet obtains the optimal asymptotic convergence rate when applied to convex problems. We further show that our approach can be extended to: (i) finding stationary points in non-convex optimization with Markovian data, and (ii) obtaining better dependence on the mixing time in temporal difference (TD) learning; in both cases, our method is completely oblivious to the mixing time. Our method relies on a novel combination of multi-level Monte Carlo (MLMC) gradient estimation together with an adaptive learning method.

翻译：我们考虑数据从马尔可夫链中抽样的随机优化问题。现有方法对此设定关键依赖于已知链的混合时间，而实际应用中该参数通常未知。我们提出首个无需知晓混合时间的优化方法，且在应用于凸问题时能获得最优渐近收敛率。进一步证明该方法可扩展至：(i) 寻找马尔可夫数据非凸优化中的驻点，以及(ii) 在时序差分学习中获取更优的混合时间依赖关系——在两种情形下，我们的方法均完全独立于混合时间。该方法基于多级蒙特卡洛梯度估计与自适应学习方法的创新性结合。