We introduce the Multiscale Experience Replay (MER) algorithm for solving a class of stochastic variational inequalities (VIs) in settings where samples are generated from a Markov chain and we have access to a memory buffer to store them. Rather than uniformly sampling from the buffer, MER utilizes a multi-scale sampling scheme to emulate the behavior of VI algorithms designed for independent and identically distributed samples, overcoming bias in the de facto serial scheme and thereby accelerating convergence. Notably, unlike standard sample-skipping variants of serial algorithms, MER is robust in that it achieves this acceleration in iteration complexity whenever possible, and without requiring knowledge of the mixing time of the Markov chain. We also discuss applications of MER, particularly in policy evaluation with temporal difference learning and in training generalized linear models with dependent data.

翻译：本文提出多尺度经验回放（MER）算法，用于解决一类随机变分不等式（VI）问题，其样本由马尔可夫链生成，且算法可利用存储这些样本的记忆缓冲区。MER并非从缓冲区均匀采样，而是采用多尺度采样方案来模拟专为独立同分布样本设计的VI算法行为，从而克服实际串行方案中的偏差并加速收敛。值得注意的是，与串行算法的标准跳采样本变体不同，MER具有鲁棒性：只要条件允许，它就能在迭代复杂度上实现加速，且无需已知马尔可夫链的混合时间。本文还探讨了MER的应用，特别是在时序差分学习的策略评估以及依赖数据下的广义线性模型训练中的应用。